Run CTS tests

Trade Federation is a continuous test framework for running tests on Android devices. To run the Compatibility Test Suite (CTS), first read the Trade Federation Overview for an explanation of Tradefed test framework.

To run a test plan:

Set up Devices Under Test (DUTs) as per Android device setup.
Ensure that you have installed the recent versions of both Android Debug Bridge (adb) and Android Asset Packaging Tool (AAPT) and added the location of those tools to the system path of your machine. For information about installing those tools, see Setting up CTS: ADB and AAPT.
Download the released CTS builds onto your Linux host machine and unzip the build to host location.
Connect at least one device. To prepare the DUT:
- Press the home button to set the device to the home screen.
- Don't use the DUT for any other tasks.
- Keep the DUT in a stationary position to avoid triggering sensor activity.
- Point the device camera at an object that can be focused.
- Do not press any keys on the device while the CTS is running. Pressing keys or touching the screen of a DUT interferes with the running tests and may lead to test failures.
Launch the CTS console cts-tradefed script from the folder where the CTS package has been unzipped. On the host's command-line shell, run:
```
./android-cts/tools/cts-tradefed
```
Run the default test plan (contains all test packages):
```
cts-tradefed > run cts
```
- If you want to improve test execution time, you can shard the tests across multiple devices. Sharding requires the host to connect at least two devices, but six or more devices are recommended for efficiency. When sharding more than 1 device:
  - For Android 9 and higher, use the command option
```
--shard-count number_of_shards
```
  - For Android 8.1 and lower, use the command option
```
--shards number_of_shards
```
- If you don't want to run the whole test suite, you can run the CTS plan of your choice from the command line:
```
run cts --plan test_plan_name
```
  To find the test plan name:
  - For Android 7.0 and higher, to see a list of test modules, enter
```
list modules
```
  - For Android 6.0 and lower, to view a list of test plans in the repository, enter
```
list plans
```
  - For Android 6.0 and lower, to view a list of test packages in the repository, enter
```
list packages
```
- For additional command options depending on CTS versions, refer to the console command reference, or under "help all" in the Tradefed console.
  
  **Note:** You may save run time in Android 7.0 (Nougat) and higher, by using the `run cts-dev` command instead of `run cts`. This command skips preconditions, device-information collection, and all system status checkers. It also runs the tests on only a single ABI. For device validation, avoid this optimization and include all preconditions and checks.
Run multiple Retry sessions until all test modules are completed and the test failure numbers are the same in the last two retry sessions.
- For Android 9 and higher, use
```
run retry --retry session_number --shard-count number_of_shards
```
  **Note:** The `--include-filter` option is not supported in the new retry command.
- For Android 7.0–8.1, use
```
run cts --retry session_number --shards number_of_shards
```
- For additional Retry command options depending on CTS version, see CTS v2 Command Console.
- To understand implementation details for CTS retry, see Trade Federation Suite Retry.
Run a retry session only on failed parameterized tests. Passed parameterized tests aren't retried.
- For Android 11 and higher, the following retry command option is enabled by default in the run cts command:
```
run retry --retry  --new-parameterized-handling
```
View test progress and results reported on the console.

Run CTS using Android Test Station

Android Test Station is a test automation tool that Android developers and test engineers can use to employ a user interface for running standard test suites. It works with the Android Compatibility Test Suite (CTS).

The tool is publicly available through ATS User Guide, and its code is open sourced in AOSP (multitest_transport, tradefed_cluster).

Run CTS for Alternate Modes

CTS 10 R4 release adds a test plan for devices with alternate modes, or for devices with more than one screen. Execute the alternate modes test plan using run cts-foldable.

Passed or failed test cases for alternate screen mode are appended with the value from display_mode, for example, testcase1[display_mode=0].

In CTS 13, the DeviceStateManager API allows test modules with the following configuration option to be run on different device states of a foldable device. Test execution is automatic in CTS based on the display folding states defined in the device without the need to run the cts-foldable test plan.

<option name="config-descriptor:metadata" key="parameter" value="all_foldable_states" />

Multi-device testing in CTS

Android 13 and higher support multi-device testing. Tests that require multiple devices automatically execute during the CTS run. CTS 13 includes some multi-device tests that trigger automatically when sharding is used. While testing doesn’t require additional changes to physical setup, virtualenv needs to be installed for tests to run correctly. For details, see Multi-Devices Suites.

Run multi-device tests

To run multi-device testing independently, use the following code:

  cts-tradefed > run cts-multidevice

All regular options are supported. To target specific devices, add --serial <serial1> --serial <serial2> and so on for the number of devices targeted.

To trigger multi-device tests automatically, use sharding, as in --shard-count 2.