A/B (seamless) system updates

Legacy A/B system updates, also known as seamless updates , ensure a workable booting system remains on the disk during an over-the-air (OTA) update. This approach reduces the likelihood of an inactive device after an update, which means fewer device replacements and device reflashes at repair and warranty centers. Other commercial-grade operating systems such as ChromeOS also use A/B updates successfully.

For more information about A/B system updates and how they work, see Partition selection (slots).

A/B system updates provide the following benefits:

  • OTA updates can occur while the system is running, without interrupting the user. Users can continue to use their devices during an OTA—the only downtime during an update is when the device reboots into the updated disk partition.
  • After an update, rebooting takes no longer than a regular reboot.
  • If an OTA fails to apply (for example, because of a bad flash), the user will not be affected. The user will continue to run the old OS, and the client is free to re-attempt the update.
  • If an OTA update is applied but fails to boot, the device will reboot back into the old partition and remains usable. The client is free to re-attempt the update.
  • Any errors (such as I/O errors) affect only the unused partition set and can be retried. Such errors also become less likely because the I/O load is deliberately low to avoid degrading the user experience.
  • Updates can be streamed to A/B devices, removing the need to download the package before installing it. Streaming means it's not necessary for the user to have enough free space to store the update package on /data or /cache.
  • The cache partition is no longer used to store OTA update packages, so there is no need to ensure that the cache partition is large enough for future updates.
  • dm-verity guarantees a device will boot an uncorrupted image. If a device doesn't boot due to a bad OTA or dm-verity issue, the device can reboot into an old image. (Android Verified Boot does not require A/B updates.)

About A/B system updates

A/B updates require changes to both the client and the system. The OTA package server, however, should not require changes: update packages are still served over HTTPS. For devices using Google's OTA infrastructure, the system changes are all in AOSP, and the client code is provided by Google Play services. OEMs not using Google's OTA infrastructure will be able to reuse the AOSP system code but will need to supply their own client.

For OEMs supplying their own client, the client needs to:

  • Decide when to take an update. Because A/B updates happen in the background, they are no longer user-initiated. To avoid disrupting users, it is recommended that updates are scheduled when the device is in idle maintenance mode, such as overnight, and on Wi-Fi. However, your client can use any heuristics you want.
  • Check in with your OTA package servers and determine whether an update is available. This should be mostly the same as your existing client code, except that you will want to signal that the device supports A/B. (Google's client also includes a Check now button for users to check for the latest update.)
  • Call update_engine with the HTTPS URL for your update package, assuming one is available. update_engine will update the raw blocks on the currently unused partition as it streams the update package.
  • Report installation successes or failures to your servers, based on the update_engine result code. If the update is applied successfully, update_engine will tell the bootloader to boot into the new OS on the next reboot. The bootloader will fallback to the old OS if the new OS fails to boot, so no work is required from the client. If the update fails, the client needs to decide when (and whether) to try again, based on the detailed error code. For example, a good client could recognize that a partial ("diff") OTA package fails and try a full OTA package instead.

Optionally, the client can:

  • Show a notification asking the user to reboot. If you want to implement a policy where the user is encouraged to routinely update, then this notification can be added to your client. If the client does not prompt users, then users will get the update next time they reboot anyway. (Google's client has a per-update configurable delay.)
  • Show a notification telling users whether they booted into a new OS version or whether they were expected to do so but fell back to the old OS version. (Google's client typically does neither.)

On the system side, A/B system updates affect the following:

  • Partition selection (slots), the update_engine daemon, and bootloader interactions (described below)
  • Build process and OTA update package generation (described in Implementing A/B Updates)

Partition selection (slots)

A/B system updates use two sets of partitions referred to as slots (normally slot A and slot B). The system runs from the current slot while the partitions in the unused slot are not accessed by the running system during normal operation. This approach makes updates fault resistant by keeping the unused slot as a fallback: If an error occurs during or immediately after an update, the system can rollback to the old slot and continue to have a working system. To achieve this goal, no partition used by the current slot should be updated as part of the OTA update (including partitions for which there is only one copy).

Each slot has a bootable attribute that states whether the slot contains a correct system from which the device can boot. The current slot is bootable when the system is running, but the other slot may have an old (still correct) version of the system, a newer version, or invalid data. Regardless of what the current slot is, there is one slot that is the active slot (the one the bootloader will boot from on the next boot) or the preferred slot.

Each slot also has a successful attribute set by the user space, which is relevant only if the slot is also bootable. A successful slot should be able to boot, run, and update itself. A bootable slot that was not marked as successful (after several attempts were made to boot from it) should be marked as unbootable by the bootloader, including changing the active slot to another bootable slot (normally to the slot running immediately before the attempt to boot into the new, active one). The specific details of the interface are defined in boot_control.h.

Update engine daemon

A/B system updates use a background daemon called update_engine to prepare the system to boot into a new, updated version. This daemon can perform the following actions:

  • Read from the current slot A/B partitions and write any data to the unused slot A/B partitions as instructed by the OTA package.
  • Call the boot_control interface in a pre-defined workflow.
  • Run a post-install program from the new partition after writing all the unused slot partitions, as instructed by the OTA package. (For details, see Post-installation).

As the update_engine daemon is not involved in the boot process itself, it is limited in what it can do during an update by the SELinux policies and features in the current slot (such policies and features can't be updated until the system boots into a new version). To maintain a robust system, the update process should not modify the partition table, the contents of partitions in the current slot, or the contents of non-A/B partitions that can't be wiped with a factory reset.

Update engine source

The update_engine source is located in system/update_engine. The A/B OTA dexopt files are split between installd and a package manager:

For a working example, refer to /device/google/marlin/device-common.mk.

Update engine logs

For Android 8.x releases and earlier, the update_engine logs can be found in logcat and in the bug report. To make the update_engine logs available in the file system, patch the following changes into your build:

These changes save a copy of the most recent update_engine log to /data/misc/update_engine_log/update_engine.YEAR-TIME. In addition to the current log, the five most recent logs are saved under /data/misc/update_engine_log/. Users with the log group ID will be able to access the file system logs.

Bootloader interactions

The boot_control HAL is used by update_engine (and possibly other daemons) to instruct the bootloader what to boot from. Common example scenarios and their associated states include the following:

  • Normal case: The system is running from its current slot, either slot A or B. No updates have been applied so far. The system's current slot is bootable, successful, and the active slot.
  • Update in progress: The system is running from slot B, so slot B is the bootable, successful, and active slot. Slot A was marked as unbootable since the contents of slot A are being updated but not yet completed. A reboot in this state should continue booting from slot B.
  • Update applied, reboot pending: The system is running from slot B, slot B is bootable and successful, but slot A was marked as active (and therefore is marked as bootable). Slot A is not yet marked as successful and some number of attempts to boot from slot A should be made by the bootloader.
  • System rebooted into new update: The system is running from slot A for the first time, slot B is still bootable and successful while slot A is only bootable, and still active but not successful. A user space daemon, update_verifier, should mark slot A as successful after some checks are made.

Streaming update support

User devices don't always have enough space on /data to download the update package. As neither OEMs nor users want to waste space on a /cache partition, some users go without updates because the device has nowhere to store the update package. To address this issue, Android 8.0 added support for streaming A/B updates that write blocks directly to the B partition as they are downloaded, without having to store the blocks on /data. Streaming A/B updates need almost no temporary storage and require just enough storage for roughly 100 KiB of metadata.

To enable streaming updates in Android 7.1, cherrypick the following patches:

These patches are required to support streaming A/B updates in Android 7.1 and later whether using Google Mobile Services (GMS) or any other update client.

Life of an A/B update

The update process starts when an OTA package (referred to in code as a payload) is available for downloading. Policies in the device may defer the payload download and application based on battery level, user activity, charging status, or other policies. In addition, because the update runs in the background, users might not know an update is in progress. All of this means the update process might be interrupted at any point due to policies, unexpected reboots, or user actions.

Optionally, metadata in the OTA package itself indicates the update can be streamed; the same package can also be used for non-streaming installation. The server may use the metadata to tell the client it's streaming so the client will hand off the OTA to update_engine correctly. Device manufacturers with their own server and client can enable streaming updates by ensuring the server identifies the update is streaming (or assumes all updates are streaming) and the client makes the correct call to update_engine for streaming. Manufacturers can use the fact that the package is of the streaming variant to send a flag to the client to trigger hand off to the framework side as streaming.

After a payload is available, the update process is as follows:

Step Activities
1 The current slot (or "source slot") is marked as successful (if not already marked) with markBootSuccessful().
2 The unused slot (or "target slot") is marked as unbootable by calling the function setSlotAsUnbootable(). The current slot is always marked as successful at the beginning of the update to prevent the bootloader from falling back to the unused slot, which will soon have invalid data. If the system has reached the point where it can start applying an update, the current slot is marked as successful even if other major components are broken (such as the UI in a crash loop) as it is possible to push new software to fix these problems.

The update payload is an opaque blob with the instructions to update to the new version. The update payload consists of the following:
  • Metadata. A relatively small portion of the update payload, the metadata contains a list of operations to produce and verify the new version on the target slot. For example, an operation could decompress a certain blob and write it to specific blocks in a target partition, or read from a source partition, apply a binary patch, and write to certain blocks in a target partition.
  • Extra data. As the bulk of the update payload, the extra data associated with the operations consists of the compressed blob or binary patch in these examples.
3 The payload metadata is downloaded.
4 For each operation defined in the metadata, in order, the associated data (if any) is downloaded to memory, the operation is applied, and the associated memory is discarded.
5 The whole partitions are re-read and verified against the expected hash.
6 The post-install step (if any) is run. In the case of an error during the execution of any step, the update fails and is re-attempted with possibly a different payload. If all the steps so far have succeeded, the update succeeds and the last step is executed.
7 The unused slot is marked as active by calling setActiveBootSlot(). Marking the unused slot as active doesn't mean it will finish booting. The bootloader (or system itself) can switch the active slot back if it doesn't read a successful state.
8 Post-installation (described below) involves running a program from the "new update" version while still running in the old version. If defined in the OTA package, this step is mandatory and the program must return with exit code 0; otherwise, the update fails.
9 After the system successfully boots far enough into the new slot and finishes the post-reboot checks, the now current slot (formerly the "target slot") is marked as successful by calling markBootSuccessful().

Post-installation

For every partition where a post-install step is defined, update_engine mounts the new partition into a specific location and executes the program specified in the OTA relative to the mounted partition. For example, if the post-install program is defined as usr/bin/postinstall in the system partition, this partition from the unused slot will be mounted in a fixed location (such as /postinstall_mount) and the /postinstall_mount/usr/bin/postinstall command is executed.

For post-installation to succeed, the old kernel must be able to:

  • Mount the new filesystem format. The filesystem type cannot change unless there's support for it in the old kernel, including details such as the compression algorithm used if using a compressed filesystem (i.e. SquashFS).
  • Understand the new partition's post-install program format. If using an Executable and Linkable Format (ELF) binary, it should be compatible with the old kernel (e.g. a 64-bit new program running on an old 32-bit kernel if the architecture switched from 32- to 64-bit builds). Unless the loader (ld) is instructed to use other paths or build a static binary, libraries will be loaded from the old system image and not the new one.

For example, you could use a shell script as a post-install program interpreted by the old system's shell binary with a #! marker at the top), then set up library paths from the new environment for executing a more complex binary post-install program. Alternatively, you could run the post-install step from a dedicated smaller partition to enable the filesystem format in the main system partition to be updated without incurring backward compatibility issues or stepping-stone updates; this would allow users to update directly to the latest version from a factory image.

The new post-install program is limited by the SELinux policies defined in the old system. As such, the post-install step is suitable for performing tasks required by design on a given device or other best-effort tasks. The post-install step is not suitable for one-off bug fixes before reboot that require unforeseen permissions.

The selected post-install program runs in the postinstall SELinux context. All the files in the new mounted partition will be tagged with postinstall_file, regardless of what their attributes are after rebooting into that new system. Changes to the SELinux attributes in the new system won't impact the post-install step. If the post-install program needs extra permissions, those must be added to the post-install context.

After reboot

After rebooting, update_verifier triggers the integrity check using dm-verity. This check starts before zygote to avoid Java services making any irreversible changes that would prevent a safe rollback. During this process, bootloader and kernel may also trigger a reboot if verified boot or dm-verity detect any corruption. After the check completes, update_verifier marks the boot successful.

update_verifier will read only the blocks listed in /data/ota_package/care_map.txt, which is included in an A/B OTA package when using the AOSP code. The Java system update client, such as GmsCore, extracts care_map.txt, sets up the access permission before rebooting the device, and deletes the extracted file after the system successfully boots into the new version.