This page outlines the usage guidelines for Developer-Powered CTS (CTS-D).
Test coverage
CTS-D, like CTS & CTS Verifier, can only enforce the following:
- All public APIs that are described in the developer SDK (developer.android.com) for a certain API level.
- All MUST requirements that are included in the Android Compatibility Definition Document (CDD) for a certain API level.
Non-MUST requirements, such as STRONGLY RECOMMENDED, SHOULD, MAY, are optional and can’t be tested using CTS.
Because all APIs and CDD requirements are tied to a specific API level, all CTS tests (CTS, CTS-D, and CTS Verifier) are tied to the same API level as their associated APIs or requirements. If a specific API is deprecated or changed, its corresponding test must be deprecated or updated.
CTS test creation rules
- A test must produce the same objective result consistently.
- A test must determine whether a device passes or fails by testing that device one time out of the box.
- Test creators must remove all possible factors that could affect test results.
- If a device needs a certain hardware condition/environment/setup, that setup must be clearly defined in the commit message. For example setup instructions, see Setting Up CTS.
- The test must not run for more than 6 hours at a time. If it needs to run for longer, please include the reasoning in your test proposal so that we can review it.
The following is an example set of test conditions for testing an app restriction:
- Wifi is stable (for a test that relies on Wifi).
- The device remains stationary during the test (or not, depending on the test).
- The device is unplugged from any power source with X percent of battery level.
- No apps, foreground services, or background services are running, except for CTS.
- The screen is off while running CTS.
- The device is NOT
isLowRamDevice
. - Battery saver / app restrictions have not been changed from the “out-of-the-box” state.
Test eligibility
We accept new tests that enforce a behavior that isn't tested by existing CTS, CTS Verifier, or CTS-D tests. Any test that checks a behavior outside the scope of our test coverage will be rejected.
CTS submission process
- Write a test proposal: An app developer submits a test proposal using Google Issue Tracker, describing the issue that has been identified and proposing a test to check for it. The proposal must include the associated CDD requirement ID. The Android team reviews the proposal.
- Develop a CTS test: After a proposal is approved, its submitter creates a CTS test on AOSP on the main (AOSP/main) branch. The Android team reviews the code.
- Publish test: Submit your CL on
AOSP/main
and then cherry-pick it to the latestandroidx-tests-dev
branch. The test is now publicly available.
CTS-D test writing guidelines
- Follow the Java Code Style Guide.
- Follow all the steps described in CTS Development.
- Add your tests to the appropriate test plan:
- Use
include-filters
to add your new tests to the CTS-D test plan:platform/cts/tools/cts-tradefed/res/config/cts-developer.xml
. - Use
exclude-filters
to exclude your new tests from the main CTS test plan:platform/cts/tools/cts-tradefed/res/config/cts-developer-exclude.xml
.
- Use
- Handle all
errorprone
warnings and suggestions inbuild_error.log
. - Rebase your changes to
head
. This includes thects-developer.xml
andcts-developer-exclude.xml
test plans. - Work with your Google engineering contact to determine whether your test case can be included in an existing CTS module. If it can't, they'll help you create a new module.
- For each new test module created, create an OWNERS file in the new test module
directory.
- Your OWNERS file should contain the following information, obtained from the Google test owner you're working with:
# Bug component: xxx
- Google test owner ldap
- In
AndroidTest.xml
, specify the following parameters. Refer to the sample files (1, 2) for examples:Instant_app
ornot_instant_app
secondary_user
ornot_secondary_user
all_foldable_states
orno_foldable_states
- To specify the correct minSDK, refer to the <uses-sdk> documentation.
- When checking in new test methods, classes, or modules, add them to the CTS-D test plan and exclude them from the main CTS test plan in the same way as for new tests.
Run your CTS-D test
Run the CTS-D test plan from the command line
using run cts --plan cts-developer
.
To run a specific test case, use run cts --include-filter "test_module_name test_name"
.
For information on running the full CTS, see Run CTS tests.
Acceptance and release
Once a test request is submitted, an internal team will review it to make sure it tests a CDD requirement or a documented API behavior. If the test is determined to be checking for a valid requirement or behavior, the team will forward this test case to a Google engineer for further review. The Google engineer will reach out to you with feedback on how the test can be improved before it can be accepted into CTS.
See Release schedule and branch information for details on the CTS release schedule.