Test suite
For a test to be a part of VTS, it must have the following setting
in Android.bp.
test_suites: ["vts"],
Additionally, adding the test to the suite general-tests allows it to be
part of a Test Mapping suite used in presubmit checks.
Test configuration
In most cases, test configuration, which is an XML file used by Trade Federation to run a VTS test, is automatically generated during the build. However, you can customize the test configuration.
Creating a customized test configuration file
Creating a new test XML file from scratch can be complicated, as it involves undestanding how the test harness works, as well as the difference between each test runner. The autogenerated test config file is designed to make this process easier.
If you must customize the test XML file, you can use the autogenerated file as a starting point.
To locate the autogenerated test config file, first
run the make command to build the config, as shown below.
$ m VtsHalUsbV1_1TargetTest
In your build out directory, you can search for the config file based on the module name, as shown below.
$ find out/ -name VtsHalUsbV1_1TargetTest.config
There can be multiple copies of the file and you can use any of them.
Copy the .config file to the directory
where the Android.bp file is.
If there's only one test module in the Android.bp file, you can
rename the XML file to AndroidTest.xml, and the build system automatically
uses that as the test module’s configuration file. Otherwise,
add a test_config attribute to the module, as shown in the
example below.
test_config: "VtsHalUsbV1_1TargetTest.xml",
Now you have a test configuration file to work with and implement the customization.
Forcing the test to run with adb root
Most VTS tests require root privilege to run. If the test configuration
file is autogenerated, you can add following attribute to Android.bp.
require_root: true,
If the test configuration file is customized, add following to the test XML file.
<target_preparer class="com.android.tradefed.targetprep.RootTargetPreparer"/>
Stopping framework during the test
Many VTS tests don't require Android framework to run, and running the
test with framework stopped allows the test to run with stability
without being affected by device flakes. If the test configuration
file is autogenerated, you can add following attribute to Android.bp.
disable_framework: true,
If the test configuration file is customized, add following to the test XML file.
<target_preparer class="com.android.tradefed.targetprep.StopServicesSetup"/>
Adding additional test arguments
Some gtest tests might require more time to run. In such cases, you can add test runner options in the XML file.
For example, the native-test-timeout setting in the following
entry allows the test to run with a timeout of 3 minutes, instead of the
default of 1 minute.
<test class="com.android.tradefed.testtype.GTest" >
<option name="native-test-device-path" value="/data/local/tmp" />
<option name="module-name" value="VtsHalNfcV1_0TargetTest" />
<option name="native-test-timeout" value="180000"/>
</test>
Requiring minimum API level
Some VTS tests can run only on devices with minimum API level. If the
test configuration file is autogenerated, you can add following
attribute to Android.bp.
test_min_api_level: 29,
If the test configuration file is customized, add the following command to the test XML file.
<object type="module_controller" class="com.android.tradefed.testtype.suite.module.MinApiLevelModuleController">
<option name="min-api-level" value="29" />
</object>
Android 12 defines ro.board.first_api_level and
ro.board.api_level properties to show the API level of the vendor images on
these devices. Combining these properties with ro.product.first_api_level,
test suites choose proper test cases for the devices.
Android 13 defines ro.vendor.api_level that is
automatically set by calculating the required vendor API level using the
ro.product.first_api_level, ro.board.first_api_level, and
ro.board.api_level properties.
ro.board.first_api_level
The ro.board.first_api_level property is the API level when the vendor images
for an SoC are first released with this property. This does not depend on the
device’s launching API level but only depends on the SoC’s first API level that
defines this value. The value is permanent for the lifetime of the SoC.
To set ro.board.first_api_level, device manufacturers can define
BOARD_SHIPPING_API_LEVEL in their device.mk file, as shown in the following
example:
# BOARD_SHIPPING_API_LEVEL sets ro.product.first_api_level to indicate
# the first api level that the device has been commercially launched on.
BOARD_SHIPPING_API_LEVEL := 23
It will automatically define the ro.board.first_api_level property to
vendor/build.prop on the device. The property is set by the vendor init
process.
ro.board.api_level
The ro.board.api_level property is the current API level of the vendor images
for an SoC. When the API level of the vendor image that has the
ro.board.first_api_level is updated, the device using the SoC must define
ro.board.api_level property with the current API level of the vendor image
instead of updating the ro.board.first_api_level. If this property is not set,
ro.board.first_api_level will be used by default.
To set the ro.board.api_level, define BOARD_API_LEVEL in the device.mk file
with the desired value.
ro.vendor.api_level
The ro.vendor.api_level property was introduced in
Android 13 to show the API level that the vendor images
are required to support. This is automatically set to the minimum of
ro.board.api_level (or ro.board.first_api_level if ro.board.api_level is
not defined) and ro.product.first_api_level. Tests for vendor implementation
that require vendor image upgrade might be excluded from the vendor requirements
for the SoC by referring to this property.
Sharding process using VTS
For Android version 10 or higher, you can perform the sharding process on multiple devices while testing with both VTS and CTS-on-GSI plans by following the instructions below.
run vts --shard-count <number of devices> -s <device serial> ...
This command splits the VTS plan into shards and runs them on multiple devices.
run cts-on-gsi --shard-count <number of devices> -s <device serial> -s ...
This command splits the CTS-on-GSI plan into shards and runs them on multiple devices.