CTS Development

Initializing your Repo client

Follow the instructions from Downloading the Source to get and build the Android source code. When issuing the repo init command, specify a specific CTS branch using -b. This ensures that your CTS changes are included in subsequent CTS releases.

Building and running CTS

Execute the following commands to build CTS and start the interactive CTS console:

cd /path/to/android/root
make cts -j32 TARGET_PRODUCT=aosp_arm64

In the CTS console, enter:

tf> run cts --plan CTS

Writing CTS tests

CTS tests use JUnit and the Android testing APIs. Review the Test your app tutorial and existing tests under the cts/tests directory. CTS tests mostly follow the same conventions used in other Android tests.

CTS runs across many production devices, so the tests must follow these rules:

  • Take into account varying screen sizes, orientations, and keyboard layouts.
  • Use only public API methods. In other words, avoid all classes, methods, and fields with the hide annotation.
  • Avoid using view layouts or relying on the dimensions of assets that may not be on some devices.
  • Don't rely upon root privileges.

Create a subplan

As an example, you can add a SubPlan.xml file in android-cts/subplans as follows:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<SubPlan version="2.0">
<Entry include="CtsSystemIntentTestCases" />
<Entry include="CtsSystemUiHostTestCases" />
<Entry include="CtsSecurityHostTestCases android.security.cts.SELinuxHostTest#testAospFileContexts" />
<Entry include="CtsSecurityHostTestCases android.security.cts.SELinuxHostTest#testAospServiceContexts" />

To run the subplan:

run cts --subplan aSubPlan

The subplan entry format is:

Include a module name as follows:
<Entry include="MODULE_NAME" />

Include a package:
<Entry include="MODULE_NAME PACKAGE_NAME" />

Include a class:

Include an individual test:

Test naming and location

Most CTS test cases target a specific class in the Android API. These tests have Java package names with a cts suffix and class names with a Test suffix. Each test case consists of multiple tests, where each test typically exercises a specific method of the class being tested. These tests are arranged in a directory structure where tests are grouped into different categories such as "widgets" or "views".

For example, the CTS test for the Java package android.widget.TextView is android.widget.cts.TextViewTest with its Java package name as android.widget.cts and its class name as TextViewTest.

  • Java package name
    The Java package name for the CTS tests is the package name of the class that the test is testing, followed by .cts. For our example, the package name would be android.widget.cts.
  • Class name
    The class name for CTS tests is the name of the class being tested with "Test" appended. For example, if a test is targeting TextView, the class name should be TextViewTest.
  • Module name (CTS v2 only)
    CTS v2 organizes tests by module. The module name is usually the second string of the Java package name (in our example, widget).

The directory structure and sample code depend on whether you are using CTS v1 or CTS v2.

CTS v1

For Android 6.0 or lower, use CTS v1. For CTS v1, the sample code is at cts/tests/tests/example.

The directory structure in CTS v1 tests looks like this:


CTS v2

For Android 7.0 or higher, use CTS v2. For details, see the sample test in Android Open Source Project (AOSP).

The CTS v2 directory structure looks like this:


New sample packages

When adding new tests, there might not be an existing directory to place your test. In those cases, you need to create the directory and copy the appropriate sample files.

CTS v1

If you're using CTS v1, refer to the example under cts/tests/tests/example and create a new directory. Also, make sure to add your new package's module name from its Android.mk to CTS_COVERAGE_TEST_CASE_LIST in cts/CtsTestCaseList.mk. build/core/tasks/cts.mk uses this makefile to combine all the tests and create the final CTS package.

CTS v2

Use the sample test /cts/tests/sample/ to quick start your new test module with the following steps:

  1. To create the test directory and copy sample files, run:
    mkdir cts/tests/module-name && cp -r cts/tests/sample/* cts/tests/module-name
  2. Navigate to cts/tests/module-name and substitute all instances of "[Ss]ample" with the recommended naming convention from above.
  3. Update SampleDeviceActivity to exercise the feature you're testing.
  4. Update SampleDeviceTest to ensure that the activity succeeds or logs its errors.

Additional directories

Other Android directories such as assets, jni, libs, and res can also be added. To add JNI code, create a directory in the root of the project next to src with the native code and an Android.mk makefile in it.

The makefile typically contains the following settings:

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := libCtsSample_jni

# don't include this package in any target
LOCAL_SRC_FILES := list of source code files

# Tag this module as a cts test artifact
LOCAL_SHARED_LIBRARIES := libnativehelper

Android.mk file

Finally, modify the Android.mk file in the root of the project to build the native code and depend on it, as shown below:

# All tests should include android.test.runner.
LOCAL_JAVA_LIBRARIES := android.test.runner

# Includes the jni code as a shared library

# Include for InstrumentationCtsTestRunner

#Tells make to look in subdirectories for more make files to include
include $(call all-makefiles-under,$(LOCAL_PATH))

Fixing or removing tests

In addition to adding new tests, you can fix or remove tests annotated with BrokenTest or KnownFailure.

Submitting your changes

When submitting CTS or VTS patches in AOSP, choose your development branch based on which API levels the patch applies to.

  • For changes that apply to multiple API levels, first develop a patch in aosp/master and then cherry-pick to the most upstream test branch. Allow the automerger to merge the changes downstream in AOSP test branches. See Release schedule and branch information for the list of branches and automerge path information.
  • For changes that are specific to a specific API level, develop or cherry-pick the changes to the correct test branch with DO NOT MERGE or RESTRICT AUTOMERGE in the commit message.

Follow the Submitting Patches workflow to contribute changes to CTS. A reviewer will be assigned to your change, and your change should be reviewed shortly.

Release schedule and branch information

CTS releases follow this schedule.

Version Branch Frequency
11 android11-tests-dev Quarterly
10 android10-tests-dev Quarterly
9 pie-cts-dev Quarterly
8.1 oreo-mr1-cts-dev Quarterly
No releases are planned for versions 8.0, 7.1, 7.0, 6.0, 5.1, 5.0, 4.4, 4.3, and 4.2.

Important dates during the release

  • End of the first week: Code freeze. Submissions to the branch until the code freeze are considered for the upcoming version of CTS. Submissions to the branch after the code freeze, or after a candidate for release is chosen, are considered for the subsequent release.
  • Second or third week: CTS is published in AOSP.

Automerge flow

CTS development branches have been set up so that changes submitted to each branch automatically merge to higher branches.

For changes to an upcoming Android version, the automerge path is:
aosp/master > <private-development-branch for Android 12>.

For changes directly to an AOSP branch, the automerge path is:
oreo-cts-dev > oreo-mr1-cts-dev > pie-cts-dev > android10-tests-dev > android11-tests-dev

If a changelist (CL) fails to merge correctly, the author of the patch is sent an email with instructions on how to resolve the conflict. In most of the cases, the author of the patch can use the instructions to skip the automerge of the conflicting CL.

If an older branch requires the change, then the patch needs to be cherry-picked from the newer branch.