Run tests (Atest)

Atest is a command line tool that allows users to build, install, and run Android tests locally, greatly speeding up test re-runs without requiring knowledge of Trade Federation test harness command line options. This page explains how to use Atest to run Android tests.

For general information on writing tests for Android, see Android Platform Testing.

For information on the overall structure of Atest, refer to the Atest Developer Guide.

For information on running tests in TEST_MAPPING files through Atest, see Running tests in TEST_MAPPING files.

To add a feature to Atest, follow the Atest Developer Workflow.

Set up your environment

To set up your Atest environment, follow the instructions in Setting up the environment, Choosing a target, and Building the code.

Basic usage

Atest commands take the following form:

atest test-to-run [optional-arguments]

Optional arguments

The following table lists the most commonly used arguments. A complete list is available through atest --help.

For more information on -b, -i and -t, see the Specify steps: build, install, or run section.

Specify tests

To run tests, specify one or more tests using one of the following identifiers:

• Module name
• Module:Class
• Class name
• Tradefed integration test
• File path
• Package name

Separate references to multiple tests with spaces, like this:

atest test-identifier-1 test-identifier-2

Module name

To run an entire test module, use its module name. Input the name as it appears in the LOCAL_MODULE or LOCAL_PACKAGE_NAME variables in that test's Android.mk or Android.bp file.

Examples:

atest FrameworksServicesTests
atest CtsVideoTestCases

Module:Class

To run a single class within a module, use Module:Class. Module is the same as described in Module name. Class is the name of the test class in the .java file, and can be the fully qualified class name or the basic name.

Examples:

atest CtsVideoTestCases:VideoEncoderDecoderTest
atest FrameworksServicesTests:ScreenDecorWindowTests
atest FrameworksServicesTests:com.android.server.wm.ScreenDecorWindowTests

Class name

To run a single class without explicitly stating a module name, use the class name.

Examples:

atest ScreenDecorWindowTests
atest VideoEncoderDecoderTest

Tradefed integration test

To run tests that are integrated directly into TradeFed (non-modules), input the name as it appears in the output of the tradefed.sh list configs command. For example:

To run the reboot.xml test:

atest example/reboot

To run the native-benchmark.xml test:

atest native-benchmark

File path

Atest supports running both module-based tests and integration-based tests by inputting the path to their test file or directory as appropriate. It also supports running a single class by specifying the path to the class's Java file. Both relative and absolute paths are supported.

Run a module

The following examples show two ways to run the CtsVideoTestCases module using a file path.

Run from Android repo-root:

atest cts/tests/video

Run from Android repo-root/cts/tests/video:

    atest .

Run a test class

The following example shows how to run a specific class within the CtsVideoTestCases module using a file path.

From Android repo-root:

    atest cts/tests/video/src/android/video/cts/VideoEncoderDecoderTest.java

Run an integration test

The following example shows how to run an integration test using a file path from Android repo-root:

    atest tools/tradefederation/contrib/res/config/example/reboot.xml

Package name

Atest supports searching for tests by package name.

Examples:

    atest com.android.server.wm
    atest com.android.uibench.janktests

Specify steps: Build, install, or run

Use the -b, -i, and -t options to specify which steps to run. If you don't specify an option, then all steps run.

• Build targets only: atest -b test-to-run
• Run tests only: atest -t test-to-run
• Install apk and run tests: atest -it test-to-run
• Build and run, but don't install: atest -bt test-to-run

Atest can force a test to skip the cleanup or teardown step. Many tests, such as CTS, clean up the device after the test is run, so trying to rerun your test with -t will fail without the --disable-teardown parameter. Use -d before -t to skip the test clean up step and test iteratively.

atest -d test-to-run
atest -t test-to-run

Run specific methods

Atest supports running specific methods within a test class. Although the whole module needs to be built, this reduces the time needed to run the tests. To run specific methods, identify the class using any of the ways supported for identifying a class (Module:Class, file path, etc) and append the name of the method:

atest reference-to-class#method1

When specifying multiple methods, separate them with commas:

atest reference-to-class#method1,method2,method3

Examples:

atest com.android.server.wm.ScreenDecorWindowTests#testMultipleDecors
atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChange,testRemoval

The following two examples show the preferred ways to run a single method, testFlagChange. These examples are preferred over only using the class name because specifying the module or the Java file location allows Atest to find the test much more quickly.

Using Module:Class:

atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChange

From Android repo-root:

atest frameworks/base/services/tests/wmtests/src/com/android/server/wm/ScreenDecorWindowTests.java#testFlagChange

Multiple methods can be run from different classes and modules:

atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChange,testRemoval ScreenDecorWindowTests#testMultipleDecors

Run multiple classes

To run multiple classes, separate them with spaces in the same way as for running multiple tests. Atest builds and runs classes efficiently, so specifying a subset of classes in a module improves performance over running the whole module.

To run two classes in the same module:

atest FrameworksServicesTests:ScreenDecorWindowTests FrameworksServicesTests:DimmerTests

To run two classes in different modules:

atest FrameworksServicesTests:ScreenDecorWindowTests CtsVideoTestCases:VideoEncoderDecoderTest

Run GTest binaries

Atest can run GTest binaries. Use -a to run these tests for all available device architectures, which in this example are armeabi-v7a (ARM 32-bit) and arm64-v8a (ARM 64-bit).

Example input test:

atest -a libinput_tests inputflinger_tests

To select a specific GTest binary to run, use a colon (:) to specify the test name, and a hashtag (#) to further specify an individual method.

For example, for the following test definition:

TEST_F(InputDispatcherTest, InjectInputEvent_ValidatesKeyEvents)

Run the following to specify the entire test:

atest inputflinger_tests:InputDispatcherTest

Or run an individual test using the following:

atest inputflinger_tests:InputDispatcherTest#InjectInputEvent_ValidatesKeyEvents

Run tests in TEST_MAPPING

Atest can run tests in TEST_MAPPING files.

Run presubmit tests implicitly

Run presubmit tests in TEST_MAPPING files in current and parent directories:

atest

Run presubmit tests in TEST_MAPPING files in /path/to/project and its parent directories:

atest --test-mapping /path/to/project

Run a specified test group

The available test groups are: presubmit(default), postsubmit, mainline-presubmit, and all.

Run postsubmit tests in TEST_MAPPING files in current and parent directories:

atest :postsubmit

Run tests from all groups in TEST_MAPPING files:

atest :all

Run postsubmit tests in TEST_MAPPING files in /path/to/project and its parent directories:

atest --test-mapping /path/to/project:postsubmit

Run mainline tests in TEST_MAPPING files in /path/to/project and its parent directories:

atest --test-mapping /path/to/project:mainline-presubmit

Run tests in subdirectories

By default, Atest only searches for tests in TEST_MAPPING files upwards (from the current or the given directory to its parent directories). If you also want to run tests in TEST_MAPPING files in the subdirectories, use --include-subdirs to force Atest to include those tests as well:

atest --include-subdirs /path/to/project

Run tests in iteration

Run tests in iteration by passing the --iterations argument. Whether it passes or fails, Atest will repeat the test until the max iteration is reached.

Examples:

By default, Atest iterates 10 times. The number of iterations must be a positive integer.

atest test-to-run --iterations
atest test-to-run --iterations 5

The following approaches make it easier to detect flaky tests:

Approach 1: Run all tests until a failure occurs or the max iteration is reached.

• Stop when a failure occurs or the iteration reaches the 10th (by default) round.

  atest test-to-run --rerun-until-failure
  

• Stop when a failure occurs or the iteration reaches the 100th round.

  atest test-to-run --rerun-until-failure 100
  

Approach 2: Run only failed tests until passed or the max iteration is reached.

• Assume test-to-run has multiple test cases and one of the tests fails. Run only the failed test 10 times (by default) or until the test passes.

  atest test-to-run --retry-any-failure
  

• Stop running the failed test when it passes or reaches the 100th round.

  atest test-to-run --retry-any-failure 100
  

Run tests on AVDs

Atest is able to run tests on a newly created AVD. Run acloud create to create an AVD and build artifacts, then use the following examples to run your tests.

Start an AVD and run tests on it:

acloud create --local-instance --local-image && atest test-to-run

Start an AVD as part of a test run:

atest test-to-run --acloud-create "--local-instance --local-image"

For more information, run acloud create --help.

Pass options to module

Atest is able to pass options to test modules. To add TradeFed command line options to your test run, use the following structure and make sure your custom arguments follow the Tradefed command line option format.

atest test-to-run -- [CUSTOM_ARGS]

Pass test module options to target preparers or test runners defined in the test config file:

atest test-to-run -- --module-arg module-name:option-name:option-value
atest GtsPermissionTestCases -- --module-arg GtsPermissionTestCases:ignore-business-logic-failure:true

Pass options to a runner type or class:

atest test-to-run -- --test-arg test-class:option-name:option-value
atest CtsVideoTestCases -- --test-arg com.android.tradefed.testtype.JarHosttest:collect-tests-only:true

For more information on test-only options, see Pass options to the modules.