Developer-Powered CTS

This page outlines the usage guidelines for Developer-Powered CTS (CTS-D).

Test coverage

CTS-D, like CTS & CTS Verifier, can only enforce the following:

  • All public APIs that are described in the developer SDK (developer.android.com) for a certain API level.
  • All MUST requirements that are included in the Android Compatibility Definition Document (CDD) for a certain API level.

Non-MUST requirements, such as STRONGLY RECOMMENDED, SHOULD, MAY, are optional and can’t be tested using CTS.

Because all APIs and CDD requirements are tied to a specific API level, all CTS tests (CTS, CTS-D, and CTS Verifier) are tied to the same API level as their associated APIs or requirements. If a specific API is deprecated or changed, its corresponding test must be deprecated or updated.

CTS test creation rules

  • A test must produce the same objective result consistently.
  • A test must determine whether a device passes or fails by testing that device one time out of the box.
  • Test creators must remove all possible factors that could affect test results.
  • If a device needs a certain hardware condition/environment/setup, that setup must be clearly defined in the commit message. For example setup instructions, see Setting Up CTS.
  • The test must not run for more than 6 hours at a time. If it needs to run for longer, please include the reasoning in your test proposal so that we can review it.

The following is an example set of test conditions for testing an app restriction:

  • Wifi is stable (for a test that relies on Wifi).
  • The device remains stationary during the test (or not, depending on the test).
  • The device is unplugged from any power source with X percent of battery level.
  • No apps, foreground services, or background services are running, except for CTS.
  • The screen is off while running CTS.
  • The device is NOT isLowRamDevice.
  • Battery saver / app restrictions have not been changed from the “out-of-the-box” state.

Test eligibility

We accept new tests that enforce a behavior that isn't tested by existing CTS, CTS Verifier, or CTS-D tests. Any test that checks a behavior outside the scope of our test coverage will be rejected.

CTS submission process

  1. Write a test proposal: An app developer submits a test proposal using Google Issue Tracker, describing the issue that has been identified and proposing a test to check for it. The proposal must include the associated CDD requirement ID. The Android team reviews the proposal.
  2. Develop a CTS test: After a proposal is approved, its submitter creates a CTS test on AOSP on the main (AOSP/main) branch. The Android team reviews the code.
  3. Publish test: Submit your CL on AOSP/main and then cherry-pick it to the latest androidx-tests-dev branch. The test is now publicly available.

CTS-D test writing guidelines

  • Follow the Java Code Style Guide.
  • Follow all the steps described in CTS Development.
  • Add your tests to the appropriate test plan:
    • Use include-filters to add your new tests to the CTS-D test plan: platform/cts/tools/cts-tradefed/res/config/cts-developer.xml.
    • Use exclude-filters to exclude your new tests from the main CTS test plan: platform/cts/tools/cts-tradefed/res/config/cts-developer-exclude.xml.
  • Handle all errorprone warnings and suggestions in build_error.log.
  • Rebase your changes to head. This includes the cts-developer.xml and cts-developer-exclude.xml test plans.
  • Work with your Google engineering contact to determine whether your test case can be included in an existing CTS module. If it can't, they'll help you create a new module.
  • For each new test module created, create an OWNERS file in the new test module directory.
    • Your OWNERS file should contain the following information, obtained from the Google test owner you're working with:
    • # Bug component: xxx
    • Google test owner ldap
  • In AndroidTest.xml, specify the following parameters. Refer to the sample files (1, 2) for examples:
    • Instant_app or not_instant_app
    • secondary_user or not_secondary_user
    • all_foldable_states or no_foldable_states
  • To specify the correct minSDK, refer to the <uses-sdk> documentation.
  • When checking in new test methods, classes, or modules, add them to the CTS-D test plan and exclude them from the main CTS test plan in the same way as for new tests.

Run your CTS-D test

Run the CTS-D test plan from the command line using run cts --plan cts-developer.

To run a specific test case, use run cts --include-filter "test_module_name test_name".

For information on running the full CTS, see Run CTS tests.

Acceptance and release

Once a test request is submitted, an internal team will review it to make sure it tests a CDD requirement or a documented API behavior. If the test is determined to be checking for a valid requirement or behavior, the team will forward this test case to a Google engineer for further review. The Google engineer will reach out to you with feedback on how the test can be improved before it can be accepted into CTS.

See Release schedule and branch information for details on the CTS release schedule.