Android 9 includes a Vendor Test Suite (VTS) infrastructure for automated testing of VTS, CTS, or other tests on partner devices running the AOSP generic system image (GSI). Previously, running these tests was a highly manual operation; the new VTS test infrastructure is designed to support automated testing multiple times a day on multiple devices.
The VTS automated testing infrastructure uses the following architecture:
When a test is triggered, the VTS automated testing infrastructure performs the following tasks:
- Fetches build artifacts and test resources from different locations:
- Partner Android Build (PAB). For the GSI, VTS framework, and some other builds.
- Local filesystem, Google Cloud Storage, or other vendor-specific build system. For partners who do not store builds in Google's cloud.
- Flashes build artifacts (from the device) and the GSI (from AOSP) onto the connected device(s).
- Runs VTS tests using local TradeFed or a TradeFed in the cloud.
- Reports test results to the VTS dashboard
The process is coordinated by the VTS host controller (HC), a machine in the lab that directs the behavior of all connected devices under test. The HC is responsible for fetching the latest builds, flashing them onto devices, and invoking tests (either locally or through the commander). It also communicates with a cloud scheduler and directs traffic between the scheduler and the TradeFed instance (or some other harness) running on the HC. For details on the host controller, see Host Controller Architecture.
Automated testing requires resources such as system builds, test files, and VTS artifacts. While it's possible to build these from source, it is easier to build them from tip-of-tree regularly then post the artifacts for download.
Partners can access automation resources using the following locations:
- Partner Android Build. Programmatic access granted on a per-account basis.
- Local filesystem (or similar). For partners who do not use the Partner Android Build.
For use in flashing the devices later, resources include build providers for
both options, extending from a single
stores the builds in local temporary directories.
Partner Android Build
In Android 8.1 and lower releases, Android partners were required to visit the Partner Android Build website (https://partner.android.com/build), navigate to their account, and fetch the latest system images through the user interface. To help partners avoid this slow and labor-intensive process, Android 9 includes support for automatically downloading these resources from PAB when the appropriate credentials are provided.
Programmatic access uses OAuth2 on Google APIs to access the required RPCs.
approach for generating OAuth2 credentials, the partner must set up a
client id/secret pair with Google. When the
PartnerAndroidBuildClient is pointed to that secret for the first
time, it opens a browser window for the user to log in to their Google
account, which generates the OAuth2 credentials needed to move forward. The
credentials (access token and refresh token) are stored locally, meaning
partners should need to login only once.
POST request for URL
Clicking a resource link in PAB sends a POST request that includes the necessary data for that resource, including:
- build id, build target
- resource name
- release candidate name and whether or not the candidate is an internal build
The POST request is received by the
buildsvc RPC, which returns a URL that can be used to
access the resource.
- For Clockwork Companion APK resources, the URL is a readable URL hosted on PAB (which is auth-protected and accessible with the appropriate OAuth2 credentials).
- For other resources, the URL is long, non-protected URL from the internal Android Build API (which expires after five minutes).
Getting the URL
To avoid cross-site request forgery, the
buildsvc RPC requires an
XSRF token to be POSTed with the other parameters. While this token makes the
process more secure, it also makes programmatic access much harder since the
required for access.
To avoid this issue, Android 9 redesigns the URL
naming scheme for all files (not just APKs) to use predictable URL names for
accessing artifact lists and artifact URLs. The PAB now uses a convenient URL
format that enables partners to download resources; HC scripts can download
those APKs easily, since the URL format is known, and HC can bypass the
XSRF/cookie issues because it does not need the
Given a directory with a list (or zip file) of artifacts, the build provider sets the relevant images based on what's in the directory. You can use the gsutil tool to copy files from Google Cloud Storage to a local directory.
After the most recent device images are downloaded to the host, those images
must be flashed onto the devices. This is done using the standard
fastboot commands and Python subprocesses,
based on the temporary file paths stored by the build providers.
- Flashing only the GSI
- Flashing individual images from the main system (e.g.,
fastboot flash boot boot.img)
- Flashing all images from the main system. Example:
fastboot flashall(using the built-in
fastboot flash(one at a time)
In Android 9, the VTS automated testing infrastructure supports only the TradeFed test harness but could be extended to support other harnesses in the future.
After the devices are prepared, you can invoke tests using one of the following options:
- When using TradeFed locally, use the
testcommand in the host controller, which takes the name of a VTS test plan (e.g.
vts-selftest) and runs the test.
- When using a TradeFed Cluster (optionally connected to MTT), use the
leasecommand in the host controller console, which looks for unfulfilled test runs.
If using TradeFedCluster, TradeFed runs locally as a remote manager. If not, the tests are invoked using Python subprocesses.
Test results are automatically reported to some VTS dashboard projects by