The Android Compatibility Test Suite (CTS) provides millions of individual tests. While necessary to run CTS frequently during the software development phase, it's possible to shorten the time needed to run these tests.
This page describes methods you can use to reduce test execution time and how to optimize hardware resources into the process.
Sharding devices
To reduce cycle time, consider running the CTS on multiple devices (sharding). To see how sharding can be used, review Run CTS tests.
Android test station
Use the Android Test Station (ATS) to employ a user interface to run standard Android test suites. This tool serves as a web interface for Trade Federation (TF), allowing you to run the CTS with minimal setup on a set of test devices as well as to establish a schedule to continuously run tests.
The Android test station supports Multi-host mode, with which a single ATS controller host can be used to manage devices and tests on multiple ATS worker hosts.
Emulator continuous run
To run the CTS continuously during the development phase, Android Virtual Devices (AVD) can be used as a substitute for hardware. Regressions of test failures can be identified early, saving much of the time needed to triage and analyze root causes. Multiple instances of the emulator can be used for sharding and can be scheduled to run continuously with the Android test station.
drawElements Quality Program (dEQP)
The
drawElements Quality Program (dEQP)
is included in the Android CTS. Called CtsDepqTestCases, this program focuses
on test coverage of Android graphics. This module accounts for almost 80% of all
test cases in Android CTS and represents 6% of total execution time.
Since the Android graphics drivers are part of Android firmware (BSP) and do not change much over the course of development, you can run this module strategically. For example, if you run CTS every two weeks (or less) during software development, based on the firmware update schedule you can exclude this module for several cycles.
One option is to run CtsDeqpTestCases separately on a set of devices, and to
then submit the CTS reports. For example, on two different hosts.
Host 1:
cts-tf > run cts --max-log-size 100 --shard-count 6 -o -m CtsDeqpTestCases
Host 2:
cts-tf > run cts --max-log-size 100 --shard-count 6 -o --exclude-filter CtsDeqpTestCases
Media test cases
Media test cases verify multimedia services such as audio, video, and the multimedia drivers. These multimedia test modules contribute the most to CTS execution time. Delays can occur when:
- Downloading media files or repeatedly playing media files during tests.
- Retrying failed test cases.
The Android CTS contains these test modules:
CtsMediaStressTestCasesCtsMediaPlayerTestCasesCtsMediaAudioTestCasesCtsVideoTestCasesCtsMediaDecoderTestCasesCtsMediaCodecTestCasesCtsMediaV2TestCases
Consider running some media tests locally or on a local server. For details, see Run CTS media tests locally.
The multimedia framework and its drivers (decoders and encoders) are part of the Android firmware (BSP). You can run this module strategically and exclude these modules for several cycles, based on the firmware update schedule.