Complete Automotive Tests in a Box (CATBox) is an open source package that provides the frameworks and tools required to streamline and execute automotive tests with minimal configuration. CATBox is scalable and compatible with the tools, infrastructure, and tests you use to test and validate your Android Automotive OS (AAOS) implementations. As a result, you can provide high-quality products to the marketplace.
CATBox design and implementation
The CATBox test suite provides the tools, frameworks, and tests needed to execute functional and performance tests on an automotive target device.
The following image illustrates a high level design of a CATBox package.
Figure 1. CATBox package
Tradefed is an open source continuous testing framework used to execute tests on Android devices. For details, see Trade Federation Overview.
Automotive test framework
CATBox is based on an Automotive test framework called Spectatio, which provides APIs for testing various applications on automotive devices. It is built on top of UI Automator, an open source testing framework, that provides a set of APIs to build UI tests that interact with user and system applications.
CATBox test suite downloads
Android 12 is the release of the development milestone code-named S. Use the following links to download CATBox test suites for Android 12:
Android 11 is the release of the development milestone code-named R. Use the following links to download CATBox test suites for Android 11:
catbox build command once the Android AOSP code is available on the
local workstation. To build the CATBox package locally:
Select Device Target and run the following command:
# Select Device Target Based on Device Architecture $ lunch <target>
To build CATBox, run the following command:
# Build CATBox $ make catbox
This command creates the
android-catbox.zipfile in the
/out/host/linux-x86/catboxdirectory, which can then be used to run tests.
The environment setup required to execute CATBox tests is similar to Setting up CTS. To set up the environment, complete each of the following tasks:
- Install ADB and AAPT.
- Install JDK. See Java Development Kit for Ubuntu.
- Increase memory limit. See Storage requirements.
CATBox setup and execution
Use the Tradefed test framework to run the tests on Android devices. Before CATBox setup, familiarize yourself with Trade Federation Overview.
To set up and run CATBox, see the sections below.
Before running a test plan, complete the following steps to set up your device:
Ensure that you have installed the recent versions of both the Android Debug Bridge (adb) and the Android Asset Packaging Tool (AAPT), as well as added the location of these tools to the system path of your machine. For details, see ADB and AAPT.
Connect at least one device and prepare the Device Under Test (DUT) as follows:
- To factory reset the automotive device, click Settings > System > Reset Options > Erase all data.
- Wait for the device to automatically restart.
Make sure adb is enabled. To confirm the device is available, run the following command:
$ adb devices
To enable root, run the following command:
# Enable Root $ adb -s DEVICE_SERIAL root
DEVICE_SERIALwith the Serial ID for the automotive device.
Connect to Wi-Fi.
Extract the CATBox package
After completing the device setup, unzip the CATBox package and go to the extracted folder:
# Extract the CATBox Package $ unzip android-catbox.zip # Navigate to extracted CATBox Package $ cd android-catbox
Execute tests in the CATBox
To run CATBox with the test plans in the CATBox package, run the following:
./tools/catbox-tradefed run commandAndExit TEST_PLAN_NAME --serial DEVICE_SERIAL
DEVICE_SERIAL with the Serial ID for the
automotive device. If you have multiple devices connected to the host machine,
DEVICE_SERIAL to distinguish them. Replace
TEST_PLAN_NAME with the name of the test plan you want
To get a complete list of available test plans, run the following:
./tools/catbox-tradefed list plans | grep -i catbox
The following table displays available test plans by type:
|Functional test plans||
|Performance test plans||
CATBox test results
Test results are saved in the
android-catbox/results/latest folder for both
functional and performance test executions.
- For test results, refer to
- For failure tests details, see
- For performance metrics results, see
Logs are saved in the path
android-catbox/logs/latest for both functional and
performance test executions.
Some tests such as Dial, Bluetooth Audio, and Contacts list jank tests require additional device setup.
The Dial test requires device setup and runtime parameters. To set up the device:
- Connect the mobile device to the Automotive head unit through Bluetooth.
- Make sure the test phone contains a SIM card to run dial tests and is connected to the mobile network.
- Dial tests use default contacts stored on the phone. Use the contacts.vcf to load contacts on the phone.
COMPANION_SERIALwith the Serial ID for the phone and run the following commands:
# Contacts can be loaded on the phone by pushing the contacts.vcf file on the phone and importing it in the Contacts application. OR # Importing contacts using adb. # Push contacts VCF file to phone $ adb -s COMPANION_SERIAL push <path-to-downloaded-contacts-vcf-file> /storage/emulated/0/Android/data/com.google.android.contacts/contacts.vcf # Grant Read Permissions $ adb -s COMPANION_SERIAL shell pm grant com.google.android.contacts android.permission.READ_EXTERNAL_STORAGE # Import contacts from the VCF file $ adb -s COMPANION_SERIAL shell am start-activity -W -t 'text/x-vcard' -d file:///storage/emulated/0/Android/data/com.google.android.contacts/contacts.vcf -a android.intent.action.VIEW com.google.android.contacts # Then accept the prompts on the UI to load the contacts.
Dial tests use small and large phone numbers for test execution. Dial tests also use the contact name and contact number for search tests. Use the following parameters to provide this information:
Parameters for Large Phone Number: --module-arg AndroidAutomotiveDialTests:instrumentation-arg:large-phone-number:=< 10 digit phone number> Parameters for Small Phone Number: --module-arg AndroidAutomotiveDialTests:instrumentation-arg:small-phone-number:=< 3 digit phone number> Parameters for Search Contact Name: --module-arg AndroidAutomotiveDialTests:instrumentation-arg:search-contact-name:=< some contact name stored on the phone > Parameters for Search Contact Number: --module-arg AndroidAutomotiveDialTests:instrumentation-arg:search-contact-number:=< contact number of the name provided in the search contact name >
Bluetooth audio test
To set up a device for the Bluetooth audio test:
- Ensure that the YouTube Music app is installed on the phone and you are logged in to an account. If not, install the app and sign-in to an account.
- Pair the phone manually to the automotive device through Bluetooth.
- Once the phone is paired, play a song on the phone and then start executing the test.
Contacts list jank test
To set up a device for the Contacts list jank test:
- Make sure the phone has several contacts so that the test can scroll through the Contacts list to collect jank metrics.
- Pair the phone manually to the automotive device through Bluetooth.
- Once the phone is paired, the contacts can be accessed on the device.