Google is committed to advancing racial equity for Black communities. See how.

Android Test Station

Android Test Station is a testing tool that Android developers and test engineers can use to employ a user interface for running standard Android test suites, such as the Android Compatibility Test Suite (CTS). This tool acts as a web interface for Trade Federation (TF), allowing you to easily run CTS on a set of test devices with minimal setup, as well as establish a schedule to continuously run tests.

Setting up Android Test Station

This section explains how to install and set up Android Test Station.

Android Test Station uses source code from these locations:

Installing Android Test Station

Follow any hardware and software requirements for the tests suites you wish to run.

The requirements for CTS are found on source.android.com.

There are no additional hardware requirements, but we recommend using a machine with 100 GB free hard drive space and 8 GB memory, which is enough to store the output files from several runs of the test suite.

Installing Docker

  1. Follow the instructions for installing Docker Community Edition (CE) on your Linux machine.

  2. Follow the post-installation steps to manage Docker as a non-root user.

  3. You might need to restart your terminal window or log out and log in again for the permission changes to take effect.

Installing Python 3.7

The Android Test Station CLI requires Python 3.7.

For Ubuntu 16.04 or earlier, you need to add the repository for Python 3.7 first, by doing one of the following:

  • Run this command:

    sudo add-apt-repository ppa:deadsnakes/ppa
    
  • Or build and install the repository from the source

To install Python 3.7, run these commands:

$ sudo apt-get update
$ sudo apt install python3.7 python3.7-distutils

Getting the Android Test Station CLI

Download the command line interface (CLI) package here.

Starting Android Test Station

Start Android Test Station with the following command:

$ ./mtt start

The first time the UI is started, it may take a few minutes to appear. The CLI displays a web URL to access the UI in a browser. By default, the web URL is localhost:8000. If needed, you can change the default port upon startup with the --port flag.

If a newer version is available, you can update to the current version. You can check the Release Notes for the latest releases.

To update to the current version, run:

$ ./mtt start --force_update

To stop the application, run:

$ ./mtt stop

To view a list of other commands, use:

$ ./mtt --help

Backing up and restoring the database

To back up the ATS database, stop the application and run the following command, which backs up the current database into a TAR file named mtt-backup.tar in your home directory:

docker run --rm --mount source=mtt-data,target=/data -v ~:/out ubuntu bash -c "cd /data && tar cvf /out/mtt-backup.tar ."

To restore, run the following command before starting the application:

docker run --rm --mount source=mtt-data,target=/data -v ~:/out ubuntu bash -c "cd /data && tar xvf /out/mtt-backup.tar"

Setup Wizard

After you install and run Android Test Station for the first time, the Setup Wizard will take you through a few steps to help you customize the tool for your environment. Any changes you make here can be reconfigured later through the Settings page.

Authenticating build channels

You can configure Android Test Station to automatically download files from the internet. To do so, click the Authenticate button and follow the steps shown.

Authenticate Build Channel

Figure 1. Authenticating a Build Channel

When the build channel is authenticated successfully, the state should turn to Authorized.

Restoring a configuration backup

If you have a configuration file backed up from another Test Station host, you can upload the file to copy any configurations modified from that host by clicking the Upload File button.

Restore Configuration Backup

Figure 2. Restoring a Configuration Backup

Importing config sets

A config set is a bundle of configs for running test suites, including related device actions, build channels, etc. Config sets are hosted in a specific Google Cloud Storage (GCS) bucket. After authenticating the GCS build channel with your Google account, you will see a list of all config sets that are available to you.

Select any config sets you would like to add to your Test Station host and click Import Selected.

Import Config Sets

Figure 3. Importing a Config Set

Including Wi-Fi settings

Some CTS tests require your device to connect to a Wi-Fi hotspot. To select your Wi-Fi network, enter the WiFi SSID and optional WiFi PSK.

Wi-Fi Settings

Figure 4. Wi-Fi hotspot settings

After completing the Setup Wizard, the page will reload with the new settings applied.

Connecting a device

USB debugging must be enabled to use a device for testing. To enable debugging:

  1. Follow the instructions in Enable developer options and debugging.

  2. If you plan to use test Android builds preloaded with custom ADB keys, put the custom .adb_key files under the ~/.android/ folder.

    The files are loaded automatically and passed to ADB to auto-enable USB debugging after the device is flashed for devices running those builds.

  3. Connect the device to the host machine using USB.

    The device will appear in the Android Test Station Devices tab within one minute after refreshing the web interface. You can also view the state of the devices on this tab.

    Connect a Device

    Figure 5. Connecting a device

The different device states are:

  • Available - The device is connected and ready to run a test.
  • Allocated - The device is connected and currently running a test. Each device can run only one test at a time, so the device must finish its current test before running a new one.
  • Fastboot - The device is in fastboot mode.
  • Unknown - Couldn't get data from the device. This usually occurs for a few seconds when a device is first connected or if there's an error with the device.

Running a test

Selecting a test

Android Test Station comes with a set of prebundled CTS configurations. To run one of these tests, go to the Test Suites tab and click Run test for the desired test.

Select a Test

Figure 6. Selecting a test

To edit or add new tests, see Adding tests.

Configuring test run

Edit the parameters to use for this specific test run. Most parameters are prepopulated with values defined in the selected test configuration.

This step can be completed using the default values, but you can change any of the parameters, such as Max Retry and Command, to suit your needs.

Configure Test Run

Figure 7. Configuring a test run

The test run parameters are:

  • Name - Name of the test suite you wish to run.
  • Run Count - Number of times this test run should be executed when scheduled. Test runs are scheduled using Trade Federation, which runs up to 20 test runs in parallel if there's capacity to do so.
  • Max Retry - Maximum number of times to retry a test run if at least one test fails. This is usually set to 4–6 retries for a full CTS run to handle flaky tests.
  • Queue Timeout - If a test run remains in the Queued state for too long, it's automatically canceled. Specify the amount of time to wait before cancellation here. The default is 24 hours.
  • Command - The command to run the test suite. You can enter additional command line arguments here. For example, run a specific module in CTS 8.1 with:

    $ cts-suite -m ShortModuleName
    
  • Retry Command - The command for retrying a test suite. You can add additional command line arguments here. For example, to retry only a specific module in CTS 8.1, use:

    $ cts --retry 0 -m ShortModuleName
    

    Retry arguments may differ from those available with the initial command, so check the supported parameters on the official site for the selected test suite.

  • Previous Test Run - If you wish to re-run a previous test run:

    • Local - If the run was started on the current host, enter the test run ID seen when viewing the details of the test run.

      Local Previous Test Run

      Figure 8. Local previous test run

    • Remote - If the run was started on a different host, upload the test results file by selecting Remote, clicking Upload Test Results File, and selecting a file from your local storage.

      Remote Previous Test Run

      Figure 9. Remote previous test run

Selecting devices

Click the checkboxes to select the devices to allocate for running the test suite. The shard count should automatically change to match the number of devices selected.

Select Devices

Figure 10. Selecting devices

All selected devices must be in the Available state to execute the test run, and they all switch to the Allocated state when the test run is executed. A test run is in the Queued state while it's waiting for devices to become available.

Adding device actions

Device actions are scripts that can be executed before each test run. Some device actions come already configured, such as flashing and rebooting. To create new device actions, see Create a new device action.

Device Actions

Figure 11. Device actions

To add a device action to a test run, click Add new action, select the checkboxes for the actions to add, and click Add Action(s). Device actions are performed sequentially. You can reorder the actions by dragging them.

Add Actions

Figure 12. Reordering actions

Setting test resources

Test resources are files required to execute a test run. For example, running CTS requires an android-cts*.zip file, and flashing a device requires you to provide the build image.

The download URL for the test suite zip file should default to the Google Drive links given to partners. You can select a different file by clicking browse. In the popup window, you can enter a file download link, use a file from an authenticated build channel, or upload a file to use from local storage.

Test Resources

Figure 13. Test resources

Below is the popup window for selecting a test resource by a web URL. You can simply enter the download url link, and click Select button to confirm the selection.

Test Resource Selector - Web URL

Figure 14. Test Resource Selector - Web URL

If you have uploaded resources to Google Grive, Google Cloud Storage (GCS), or other channels, you can also navigate to the specific channel's tab and select resources there. Here is an example for selecting a resource from google drive.

Test Resource Selector - Google Drive

Figure 15. Test Resource Selector - Google Drive

In addition to just selecting files, wildcard characters are also supported in Filename field. The documentation can be found here.

Test Resource Selector - Wildcard Pattern Support

Figure 16. Test Resource Selector - Wildcard pattern support

You can also select a file in your workstation as a test resource using local file storage.

Test Resource Selector - Local File Store

Figure 17. Test Resource Selector - Local file store

Starting a test run

After you enter the information needed for the test run, click Start Test Run. If all of the information is valid, the test run starts, and you're redirected to a page to view the details and progress of the test run.

Start Test Run

Figure 18. Starting a test run

Creating a test plan

Test plans are used to create test runs on a periodic schedule. For example, running CTS 9.0 every day at 5 PM. To create a new test plan, click Create a new test plan.

Create Test Plan

Figure 19. Creating a test plan

Configure test plan

Enter the name of the test plan and any labels you want to add. Then select a schedule to use.

  • Manual - The test plan will only create test runs when a user clicks on Run test plan in the test plan list page.
  • Periodic - The test plan will automatically schedule test runs on the periodic schedule selected.
  • Custom - The test plan will autmatically schedule test runs based on the cron expression entered.

Configure Test Plan

Figure 20. Configuring a test plan

Add test suites

Add tests suites you want to be scheduled by the test plan by clicking + Add test run configuration. Select a test suite from the Name dropdown and click Next step. Then select the devices you would like to run the test on and click Add Configuration. You can add multiple configurations for each test plan.

Configure Test Run

Figure 21. Configuring a test run

Add device actions

Add the device actions you want to be executed before each test run. See Adding device actions for more details.

Add Device Actions

Figure 22. Adding device actions

Set test resources

Adding test resources to test plans is the same as adding them to individual test runs. See Setting test resources for more details.

Set Test Resources

Figure 23. Setting test resources

Viewing test runs

Test run list

View the list of scheduled test runs on the Test Runs page. Click View to see more details about a test run.

You can also filter the list by entering a string into the filter bar and pressing the Enter key. You can use multiple filters by separating them with a comma. The filter returns all rows that contain the exact text (no substring matching) in any column, excluding Status and Created.

An empty filter returns all rows. There's currently no way to filter for rows with empty values.

Test Run List

Figure 24. Test run list

Test run details

You can view the details of a test run here, such as the status, logs, and results.

Test Run Details

Figure 25. Test run details

Test run status

The progress for a test run is shown in the Status section. If there's a related message, such as download progress, cancellation reason, or error message, it's shown here as well.

Test Run Status

Figure 26. Test run status

The test run states are:

  • Pending - Required resources are being downloaded.
  • Queued - The test is ready to be run when a device becomes available.
  • Running - The test is running on an allocated device.
  • Completed - The test has completed and reported its results.
  • Canceled - The test was canceled by the user or timed out while trying to find available devices.
  • Error - An error occurred that prevented the test from running.

Canceling a test run

If the test run hasn't completed, you can cancel it by clicking Cancel and then clicking Yes in the confirmation dialog. Test runs are also automatically canceled if they remain in the Queued state longer than the queue_timeout_seconds field. Canceling a test run while in a Running state may take a few minutes to take effect.

Cancel Test Run

Figure 27. Canceling a test run

Test run results

After a test run has finished, results will be collected and displayed. You can view additional details by clicking the arrow for each run. Click View Output Files to see the test artifacts collected, such as test_result.xml and test_result_failures.html.

Test Run Results

Figure 28. Test run results

You can view live host and Tradefed logs in the Logs tab.

Test Run Logs

Figure 29. Logs tab

The results for individual modules are on the Test Results tab.

Test Results Tab

Figure 30. Test Results tab

You can download the files used as test resources by clicking Open in the Test Resources tab.

Test Resources Tab

Figure 31. Test Resources tab

To see the details of the test run, such as create_time, go to the Config tab.

Test Config Tab

Figure 32. Config tab

Advanced features

Managing config files

Android Test Station uses configuration files written in YAML to load predefined options such as tests, build channels, and device actions. An example config file for some options is shown below.

// example_file.yaml
tests:
- id : android.cts.9_0.arm
  name: CTS 9.0 (ARM)
  test_resource_defs:
  - name: android-cts.zip
    default_download_url: https://dl.google.com/dl/android/cts/android-cts-9.0_r7-linux_x86-arm.zip
    test_resource_type: TEST_PACKAGE
  command: cts
  env_vars:
  - name: TF_PATH
    value: ${TF_WORK_DIR}/android-cts/tools:${TF_WORK_DIR}/android-cts/testcases
  - name: LD_LIBRARY_PATH
    value: ${TF_WORK_DIR}/android-cts/lib:${TF_WORK_DIR}/android-cts/lib64
  setup_scripts:
  output_file_patterns:
  - android-cts/logs/latest/.*
  - android-cts/results/latest/.*\.html
  - android-cts/results/latest/compatibility_result\..*
  - android-cts/results/latest/logo.png
  - android-cts/results/latest/test_result.xml
  result_file: test_result.xml
  java_properties:
  - name: CTS_ROOT
    value: ${TF_WORK_DIR}
  context_file_dir: android-cts/results/
  context_file_pattern: '[\d_\.]+\.zip'
  retry_command_line: retry --retry 0
  runner_sharding_args: --shard-count ${TF_SHARD_COUNT}


build_channels:
- id: google_drive
  name: Google Drive
  provider_name: Google Drive

device_actions:
- id: flash
  name: Flash
  test_resource_defs:
  - name: bootloader.img
    test_resource_type: DEVICE_IMAGE
  - name: radio.img
    test_resource_type: DEVICE_IMAGE
  - name: img.zip
    test_resource_type: DEVICE_IMAGE
  tradefed_target_preparers:
  - class_name: com.android.tradefed.targetprep.RunHostCommandTargetPreparer
    option_values:
    - name: work-dir
      values:
      - ${TF_WORK_DIR}
    - name: host-setup-command
      values:
      - adb -s $SERIAL reboot-bootloader
      - fastboot -s $SERIAL flash bootloader bootloader.img
      - fastboot -s $SERIAL flash radio radio.img
      - fastboot -s $SERIAL reboot-bootloader
      - fastboot -s $SERIAL -w update img.zip
      - adb -s $SERIAL wait-for-device
    - name: host-cmd-timeout
      values:
      - 10m

When you set up your Android Test Station instance, you can share your configuration with other users by exporting it as a file. To do this, go to the Settings page and click Export in the top right.

Config File Management

Figure 33. Configuration file management

After your configuration file is downloaded, share the file with other users. They can add the configuration file to their Android Test Station instance by clicking Import and selecting the config file.

Creating a new device action

Device actions are used for automating the device setup process. Actions are scripts executed on each device the test is running on before each test run, including before retries. To view a list of available device actions, go to the Settings page and click the Device Actions tab. Several device actions come already configured, such as rebooting and flashing.

Device Actions tab

Figure 34. Device Actions tab

Adding a new device action

  1. Click New device action.

    New Device Action button

    Figure 35. New device action button

  2. Enter a name and description.

    Device Action name

    Figure 36. Device action name

  3. Click Add Target Preparer.

  4. Enter the Trade Federation Target Preparer full class name, for example, com.android.tradefed.targetprep.RunHostCommandTargetPreparer.

    Add Target Preparer

    Figure 37. Adding a target preparer

    A list of available target preparers can be found in the com.android.tradefed.targetprep reference.

    Target Preparer List

    Figure 38. Target Preparer list

  5. Add any options to use with the target preparer. To view available options, check targetprep for the source code for each target preparer in AOSP:

    Action Option Example

    Figure 39. Action option example

  6. To add an option, click Add Target Preparer Option and enter the required values.

    Action Command Example

    Figure 40. Action command example

  7. Define the test resources that are needed to execute the device action, for example, build images for flashing. To add a resource definition, click Add Test Resource and enter the required files. If you know where your files are located, you can provide a default download URL by clicking browse.

    Action Test Resources

    Figure 41. Action test resources

  8. Click Update.

    Action Save Changes

    Figure 42. Action save changes

Managing tests

Editing a test

To edit a saved test, go to the Tests page and click Edit on the row of the test you want to modify. After changing the test configuration, click Update.

Edit a test

Figure 43. Editing a test

Adding a new test

To add a new test, go to the Tests page and click Create a New Test. Enter the appropriate information and click Create.

Create a test

Figure 44. Creating a test

Copy a test

Figure 45. Copying a test

Adding proxy server settings

If your network environment requires using proxy servers to access the internet, change the settings to use those proxy servers.

  1. Go to the Settings menu in Android Test Station and look for the Proxy Settings section in the General Settings tab.

    Proxy Server Settings

    Figure 46. Proxy server settings

  2. Enter the required proxy information and click Update.

  3. After the settings are successfully updated, restart your Android Test Station instance to apply the new proxy settings:

    $ ./mtt restart
    

Support

Bug Reporting

Your contribution to Android Test Station helps improve the development of the tool, and we want your input! See the ATS release notes for details on the latest release. To report bugs or offer suggestions, please file a bug report. Partners should report bugs or suggestions via their partner channels.

Privacy Policy

Build Channels

Authorizing build channels to access other services (for example, Google Drive, Google Cloud Storage) allows Android Test Station to download files from those services. Android Test Station downloads files to be used as test resource files based on your test run configurations.

Test Run Actions

Authorizing test run actions to access other services allows Android Test Station to upload files to those services. Android Test Station uploads files based on the test run actions defined in your test run configurations (typically used for uploading the results of a test run).