Test Mapping

This is a brief introduction of Test Mapping and an explanation of how to get started configuring tests easily in the Android Open Source Project (AOSP).

What is Test Mapping?

Test Mapping is a Gerrit-based approach that allows developers to create pre- and post-submit test rules directly in the Android source tree and leave the decisions of branches and devices to be tested to the test infrastructure itself. Test Mapping definitions are JSON files named TEST_MAPPING that can be placed in any source directory.

Atest can use the TEST_MAPPING files to run presubmit tests in the associated directories. With Test Mapping, you can add the same set of tests to presubmit checks with a simple change inside the Android source tree.

See these examples:

Add presubmit tests to TEST_MAPPING for services.core

Add presubmit and postsubmit tests to TEST_MAPPING for startop/iorap

Defining test groups

Test Mapping groups tests via a test group. The name of a test group can be any string. For example, presubmit can be for a group of tests to run when validating changes. And postsubmit tests can be used to validate the builds after changes are merged.

Packaging build script rules

In order for the Trade Federation Test Harness to run Test Mapping's test modules for a given build, these modules must have test_suite set for Soong or LOCAL_COMPATIBILITY_SUITE set for Make to one of these two suites:

  • device-tests - built against a specific device CPU
  • general-tests - built against any application binary interface (ABI)

When in doubt, put gtests in device-tests and APK tests in general-tests.

Examples:

Android.bp: test_suites: ["device-tests"],
Android.mk: LOCAL_COMPATIBILITY_SUITE := device-tests

Creating Test Mapping files

For the directory requiring test coverage, simply add a TEST_MAPPING JSON file resembling the example below. These rules will ensure the tests run in presubmit checks when any files are touched in that directory or any of its subdirectories.

Following an example

Here is a sample TEST_MAPPING file:

{
  "presubmit": [
    {
      "name": "CtsWindowManagerDeviceTestCases",
      "options": [
        {
          "include-annotation": "android.platform.test.annotations.RequiresDevice"
        }
      ]
    }
  ],
  "postsubmit": [
    {
      "name": "CtsWindowManagerDeviceTestCases"
    }
  ],
  "imports": [
    {
      "path": "frameworks/base/services/core/java/com/android/server/am"
    }
  ]
}

Setting attributes

In the above example, presubmit and postsubmit are the names of each test group. Note that a test run for postsubmit will automatically include all tests in the presubmit group. See Defining test groups for more information about test groups.

The name of the test module or Trade Federation integration test name (resource path to the test XML file, e.g., uiautomator/uiautomator-demo) can be set in the value of the name attribute. Note the name field cannot use class name or test method name. To narrow down the tests to run, you can use options such as include-filter here. See (include-filter sample usage).

The imports attribute allows you to include tests in other TEST_MAPPING files without copying the content. Note that the TEST_MAPPING files in the parent directories of the imported path will also be included.

The options attribute contains additional TradeFed command line options. In the above example, only tests with annotation Presubmit will run in presubmit; all tests will run in postsubmit.

To get a complete list of available options for a given test, run:

tradefed.sh run commandAndExit [test_module] --help

Refer to TradeFed Option Handling for more details about how options work.

Running tests with Atest

To execute the presubmit test rules locally:

  1. Go to the directory containing the TEST_MAPPING file.
  2. Run the command:
atest

All presubmit tests configured in the TEST_MAPPING files of the current directory and its parent directorys are run. Atest will locate and run two tests for presubmit (A and B).

This is the simplest way to run presubmit tests in TEST_MAPPING files in the current working directory (CWD) and parent directories. Atest will locate and use the TEST_MAPPING file in CWD and all of its parent directories, unless a TEST_MAPPING file has inherit_parent set to false.

Structuring source code

The following example shows how TEST_MAPPING files can be configured across the source tree.

src
├── project_1
│   └── TEST_MAPPING
├── project_2
│   └── TEST_MAPPING
└── TEST_MAPPING

Content of src/TEST_MAPPING:

{
  "presubmit": [
    {
      "name": "A"
    }
  ]
}

Content of src/project_1/TEST_MAPPING:

{
  "presubmit": [
    {
      "name": "B"
    }
  ],
  "postsubmit": [
    {
      "name": "C"
    }
  ],
  "other_group": [
    {
      "name": "X"
    }
  ]}

Content of src/project_2/TEST_MAPPING:

{
  "presubmit": [
    {
      "name": "D"
    }
  ],
  "import": [
    {
      "path": "src/project_1"
    }
  ]}

Specifying target directories

You can specify a target directory to run tests in TEST_MAPPING files in that directory. The following command will run two tests (A, B).

atest --test-mapping src/project_1

Running postsubmit test rules

You can also use this command to run the postsubmit test rules defined in TEST_MAPPING in src_path (default to CWD) and its parent directories:

atest [--test-mapping] [src_path]:postsubmit

Identifying test groups

You can specify test groups in the Atest command. Note that presubmit tests are part of postsubmit tests, as well. The following command will run all postsubmit tests related to files in directory src/project_1, which are three tests (A, B, C).

Or you can use :all to run all tests regardless of group. The following command runs four tests (A, B, C, X):

atest --test-mapping src/project_1:all

Including subdirectories

By default, running tests in TEST_MAPPING with Atest will run only presubmit tests configured in the TEST_MAPPING file in CWD (or given directory) and its parent directories. If you want to run tests in all TEST_MAPPING files in the sub-directories, use the option --include-subdir to force Atest to include those tests too.

atest --include-subdir

Without the --include-subdir option, Atest will run only test A. With the --include-subdir option, Atest will run two tests (A, B).