Test setup

Test suite

For a test to be a part of VTS, it must have the following setting in Android.bp.

test_suites: ["vts"],

Additionally, adding the test to the suite general-tests allows it to be part of a Test Mapping suite used in presubmit checks.

Test configuration

In most cases, test configuration, which is an XML file used by Trade Federation to run a VTS test, is automatically generated during the build. However, you can customize the test configuration.

Create a customized test configuration file

Creating a new test XML file from scratch can be complicated, as it involves undestanding how the test harness works, as well as the difference between each test runner. The autogenerated test config file is designed to make this process easier.

If you must customize the test XML file, you can use the autogenerated file as a starting point.

To locate the autogenerated test config file, first run the make command to build the config, as shown below.

$ m VtsHalUsbV1_1TargetTest

In your build out directory, you can search for the config file based on the module name, as shown below.

$ find out/ -name VtsHalUsbV1_1TargetTest.config

There can be multiple copies of the file and you can use any of them. Copy the .config file to the directory where the Android.bp file is.

If there's only one test module in the Android.bp file, you can rename the XML file to AndroidTest.xml, and the build system automatically uses that as the test module’s configuration file. Otherwise, add a test_config attribute to the module, as shown in the example below.

test_config: "VtsHalUsbV1_1TargetTest.xml",

Now you have a test configuration file to work with and implement the customization.

Force the test to run with adb root

Most VTS tests require root privilege to run. If the test configuration file is autogenerated, you can add following attribute to Android.bp.

require_root: true,

If the test configuration file is customized, add following to the test XML file.

<target_preparer class="com.android.tradefed.targetprep.RootTargetPreparer"/>

Stop framework during the test

Many VTS tests don't require Android framework to run, and running the test with framework stopped allows the test to run with stability without being affected by device flakes. If the test configuration file is autogenerated, you can add following attribute to Android.bp.

disable_framework: true,

If the test configuration file is customized, add following to the test XML file.

<target_preparer class="com.android.tradefed.targetprep.StopServicesSetup"/>

Add additional test arguments

Some gtest tests might require more time to run. In such cases, you can add test runner options in the XML file.

For example, the native-test-timeout setting in the following entry allows the test to run with a timeout of 3 minutes, instead of the default of 1 minute.

   <test class="com.android.tradefed.testtype.GTest" >
       <option name="native-test-device-path" value="/data/local/tmp" />
       <option name="module-name" value="VtsHalNfcV1_0TargetTest" />
       <option name="native-test-timeout" value="180000"/>
   </test>

Require minimum API level

Some VTS tests can run only on devices with minimum API level. If the test configuration file is autogenerated, you can add following attribute to Android.bp.

min_shipping_api_level: 29,

or

vsr_min_shipping_api_level: 202404,

If the test configuration file is customized, add the following command to the test XML file.

   <object type="module_controller" class="com.android.tradefed.testtype.suite.module.ShippingApiLevelModuleController">
       <option name="min-api-level" value="29" />
   </object>

or

   <object type="module_controller" class="com.android.tradefed.testtype.suite.module.ShippingApiLevelModuleController">
       <option name="vsr-min-api-level" value="202404" />
   </object>

API level properties

Android 12 defines ro.board.first_api_level and ro.board.api_level properties to show the API level of the vendor images on these devices. Combining these properties with ro.product.first_api_level, test suites choose proper test cases for the devices.

Android 13 defines ro.vendor.api_level that is automatically set by calculating the required vendor API level using the ro.product.first_api_level, ro.board.first_api_level, and ro.board.api_level properties.

For more details, see Vendor API level.

ro.board.first_api_level

The ro.board.first_api_level property is the API level when the vendor images for an SoC that is qualified for vendor freeze are first released with this property. This does not depend on the device's launching API level but only depends on the SoC's first API level that defines this value. The value is permanent for the lifetime of the SoC.

To set ro.board.first_api_level, device manufacturers can define BOARD_SHIPPING_API_LEVEL in their device.mk file, as shown in the following example:

  # BOARD_SHIPPING_API_LEVEL sets ro.product.first_api_level to indicate
  # the first api level that the device has been commercially launched on.
  BOARD_SHIPPING_API_LEVEL := 23

It will automatically define the ro.board.first_api_level property to vendor/build.prop on the device. The property is set by the vendor init process.

ro.board.api_level

The ro.board.api_level property is the current vendor API level of the vendor images that has YYYYMM format in which the vendor API was frozen. It is automatically set by the build system.

ro.vendor.api_level

The ro.vendor.api_level property was introduced in Android 13 to show the API level that the vendor images are required to support. This is automatically set to the ro.product.first_api_level, or ro.board.api_level if ro.board.first_api_level is present and ro.board.api_level is set to an earlier API level than ro.product.first_api_level. The version will be substitute with the corresponding vendor API level if it is set to the version from ro.product.first_api_level which is greater than or equal to 35. Tests for vendor implementation that require vendor image upgrade might be excluded from the vendor requirements for the SoC by referring to this property.

Sharding process using VTS

For Android version 10 or higher, you can perform the sharding process on multiple devices while testing with both VTS and CTS-on-GSI plans by following the instructions below.

run vts --shard-count <number of devices> -s <device serial> ...

This command splits the VTS plan into shards and runs them on multiple devices.

run cts-on-gsi --shard-count <number of devices> -s <device serial> -s ...

This command splits the CTS-on-GSI plan into shards and runs them on multiple devices.