Linux-stable merges

Every day a significant number of changes are committed to the upstream Linux kernel. These changes are generally not evaluated for security impact, but many of them have the potential to impact the security of the kernel. Evaluating each of these changes for security impact is an expensive and likely unfeasible operation. Instead a more sustainable and maintainable approach is to regularly sync changes with the upstream Linux kernel.

It is recommended to regularly update devices with newer Long Term Supported (LTS) kernels. Regular LTS updates can help address potential unrecognized security vulnerabilities, such as this Project Zero report from earlier in 2019, prior to public disclosure or discovery by malicious actors.

Prerequisites

  • Android common kernel branch (from AOSP)
  • An LTS merge staging branch for the target device kernel
  • Device kernel release branch
  • Git repo
  • Kernel build toolchain

Merge with LTS changes

Merging LTS changes
Figure 1: Merging LTS changes

The following steps outline the typical steps for an LTS merge.

  • Back-merge the target kernel release branch into the -LTS staging branch
  • Locally merge linux-stable or Android common into the -LTS staging branch
  • Resolve merge conflicts (consult area/code owners as needed)
  • Build locally and perform sanity/unit testing (see testing section below)
  • Upload and merge Android common changes to LTS staging branch
  • Test thoroughly using -LTS staging branch (see testing section below)
  • Review test results
  • Address any regressions, bisect merge as needed
  • Merge -LTS staging branch into main device kernel release branch
  • Create new Android build for your devices that includes the staging LTS kernel
  • Compile release build/ROM with new kernel

Example of merging with LTS.

Merge android-4.9 into main (via LTS staging) and checkout and sync LTS staging branch:

repo init -b <Device kernel LTS staging branch>  # init
repo sync
git checkout -b lts <Device kernel LTS staging branch>
git merge <Device kernel release branch>         # back-merge
git commit

At this point it is best to push the back-merge to your source remote before continuing. After that, merge Android common into LTS staging.

git merge -X patience android-4.9-q            # LTS merge

Resolve merge conflicts

In most cases, there are conflicts between the Android common kernel and the -LTS staging branch. Resolving merge conflicts during LTS merges can be challenging so below are some helpful tips on addressing them.

Incremental merging

If a significant amount of time has passed since a device kernel was updated with LTS, there is a good chance that there have been many (>50) stable releases since the last merged update was released upstream. The best way to address this is to slowly catch up by merging in a smaller number of releases at a time (<=5 minor versions), while testing at each step of the way.

For example, if device kernel version sublevel is 4.14.100 and the upstream stable sublevel is 4.14.155, it is best to merge in small increments to ensure a reasonable volume of changes can be adequately reviewed and tested.

In general, we have found that working incrementally in batches of <=5 minor releases per merge ensures a more manageable set of patches.

Testing

Quick boot test

To perform a quick boot test you must first merge LTS changes locally and build the kernel. The following steps explain the quick boot test process.

Connect the target device to your computer using a USB cable and push .ko to the device using Android Debug Bridge (adb).

adb root
adb disable-verity
adb reboot
(wait for device boot-to-home)
adb root
adb remount
adb push *.ko vendor/lib/modules/
adb reboot bootloader

Boot dtbo and sideload the kernel image.

fastboot boot --header-version=2 Image.lz4 (redo again if device rebooted)

Check the /dev/kmsg log for an errors.

adb shell
su
cat /dev/kmsg (inspect kernel log for obvious new errors)

Android tests

First build the -userdebug image locally with the new LTS kernel and modules.

Check the /dev/kmsg for any errors and confirm there are none before continuing. Test the following things to ensure everything is acting as expected.

  • Wi-Fi speed
  • Chrome browser
  • Image and video capture with camera app
  • YouTube video playback with built-in speakers and Bluetooth headset
  • Calls over carrier network
  • Video call over Wi-Fi

Automated testing suites

The final verification to ensure the product image doesn't regress is performed using the tests suites available through the vendor test suite (VTS) and automated stability stress testing.