Trade Federation is a continuous test framework for running tests on Android devices. To run the Compatibility Test Suite (CTS), first read the Trade Federation Overview for an explanation of Tradefed test framework.
To run a test plan:
- Set up Devices Under Test (DUTs) as per Android device setup.
- Ensure that you have installed the recent versions of both Android Debug Bridge (adb) and Android Asset Packaging Tool (AAPT) and added the location of those tools to the system path of your machine. For information about installing those tools, see Setting up CTS: ADB and AAPT.
- Download the released CTS builds onto your Linux host machine and unzip the build to host location.
- Connect at least one device. To prepare the DUT:
- Press the home button to set the device to the home screen.
- Don't use the DUT for any other tasks.
- Keep the DUT in a stationary position to avoid triggering sensor activity.
- Point the device camera at an object that can be focused.
- Do not press any keys on the device while the CTS is running. Pressing keys or touching the screen of a DUT interferes with the running tests and may lead to test failures.
Launch the CTS console cts-tradefed script from the folder where the CTS package has been unzipped. On the host's command-line shell, run:
Run the default test plan (contains all test packages):
cts-tradefed > run cts
If you want to improve test execution time, you can shard the tests across multiple devices. Sharding requires the host to connect at least two devices, but six or more devices are recommended for efficiency. When sharding more than 1 device:
For Android 9 and higher, use the command option
For Android 8.1 and lower, use the command option
If you don't want to run the whole test suite, you can run the CTS plan of your choice from the command line:
run cts --plan test_plan_name
To find the test plan name:
For Android 7.0 and higher, to see a list of test modules, enter
For Android 6.0 and lower, to view a list of test plans in the repository, enter
For Android 6.0 and lower, to view a list of test packages in the repository, enter
For additional command options depending on CTS versions, refer to the console command reference, or under "help all" in the Tradefed console.
Run multiple Retry sessions until all test modules are completed and the test failure numbers are the same in the last two retry sessions.
For Android 9 and higher, use
run retry --retry session_number --shard-count number_of_shards
For Android 7.0–8.1, use
run cts --retry session_number --shards number_of_shards
For additional Retry command options depending on CTS version, see CTS v2 Command Console.
To understand implementation details for CTS retry, see Trade Federation Suite Retry.
Run a retry session only on failed paratermerized tests. Passed parameterized tests aren't retried.
For Android 11 and higher, the following retry command option is enabled by default in the
run retry --retry
View test progress and results reported on the console.
Run CTS using Android Test Station
Android Test Station is a test automation tool that Android developers and test engineers can use to employ a user interface for running standard test suites. It works with the Android Compatibility Test Suite (CTS).
Run CTS for Alternate Modes
CTS 10 R4 release adds a test plan for devices with alternate modes, or for
devices with more than one screen. Execute the alternate modes test plan using
Passed or failed test cases for alternate screen mode are appended with the
display_mode, for example,
In CTS 13, the DeviceStateManager API allows test modules with the following
configuration option to be run on different device states of a foldable device.
Test execution is automatic in CTS based on the display folding states
defined in the device without the need to run the
cts-foldable test plan.
<option name="config-descriptor:metadata" key="parameter" value="all_foldable_states" />
Multi-device testing in CTS
Android 13 and higher support multi-device testing.
Tests that require multiple devices automatically execute during the CTS run.
CTS 13 includes some multi-device tests that trigger automatically when sharding
is used. While testing doesn’t require additional changes to physical setup,
virtualenv needs to be installed
for tests to run correctly. For details,
Run multi-device tests
To run multi-device testing independently, use the following code:
cts-tradefed > run cts-multidevice
All regular options are supported. To target specific devices, add
--serial <serial1> --serial <serial2> and so on for the number of devices
To trigger multi-device tests automatically, use sharding, as in