Complete Automotive Tests in a Box (CATBox) is an open source package that provides the frameworks and tools required to streamline and execute automotive tests with minimal configuration. CATBox is scalable and compatible with the tools, infrastructure, and tests you use to test and validate your Android Automotive OS (AAOS) implementations. As a result, you can provide high-quality products to the marketplace.
CATBox design and implementation
The CATBox test suite provides the tools, frameworks, and tests needed to execute functional and performance tests on an automotive target device.
The following image illustrates a high level design of a CATBox package.
Figure 1. CATBox package
Tradefed
Tradefed is an open source continuous testing framework used to execute tests on Android devices. For details, see Trade Federation Overview.
Automotive test framework
CATBox is based on an Automotive test framework called Spectatio, which provides APIs for testing various applications on automotive devices. It is built on top of UI Automator, an open source testing framework, that provides a set of APIs to build UI tests that interact with user and system applications.
CATBox test suite downloads
Android 12
Android 12 is the release of the development milestone code-named S. Use the following links to download CATBox test suites for Android 12:
Android 11
Android 11 is the release of the development milestone code-named R. Use the following links to download CATBox test suites for Android 11:
Build CATBox
Use the catbox
build command once the Android AOSP code is available on the
local workstation. To build the CATBox package locally:
Select Device Target and run the following command:
# Select Device Target Based on Device Architecture $ lunch <target>
To build CATBox, run the following command:
# Build CATBox $ make catbox
This command creates the
android-catbox.zip
file in the/out/host/linux-x86/catbox
directory, which can then be used to run tests.
Environment setup
The environment setup required to execute CATBox tests is similar to Setting up CTS. To set up the environment, complete each of the following tasks:
- Install ADB and AAPT.
- Install JDK. See Java Development Kit for Ubuntu.
- Increase memory limit. See Storage requirements.
CATBox setup and execution
Use the Tradefed test framework to run the tests on Android devices. Before CATBox setup, familiarize yourself with Trade Federation Overview.
To set up and run CATBox, see the sections below.
Device setup
Before running a test plan, complete the following steps to set up your device:
Ensure that you have installed the recent versions of both the Android Debug Bridge (adb) and the Android Asset Packaging Tool (AAPT), as well as added the location of these tools to the system path of your machine. For details, see ADB and AAPT.
Connect at least one device and prepare the Device Under Test (DUT) as follows:
- To factory reset the automotive device, click Settings > System > Reset Options > Erase all data.
- Wait for the device to automatically restart.
Make sure adb is enabled. To confirm the device is available, run the following command:
$ adb devices
To enable root, run the following command:
# Enable Root $ adb -s DEVICE_SERIAL root
Replace
DEVICE_SERIAL
with the Serial ID for the automotive device.Connect to Wi-Fi.
Extract the CATBox package
After completing the device setup, unzip the CATBox package and go to the extracted folder:
# Extract the CATBox Package
$ unzip android-catbox.zip
# Navigate to extracted CATBox Package
$ cd android-catbox
Execute tests in the CATBox
To run CATBox with the test plans in the CATBox package, run the following:
./tools/catbox-tradefed run commandAndExit TEST_PLAN_NAME --serial DEVICE_SERIAL
Replace DEVICE_SERIAL
with the Serial ID for the
automotive device. If you have multiple devices connected to the host machine,
use DEVICE_SERIAL
to distinguish them. Replace
TEST_PLAN_NAME
with the name of the test plan you want
to run.
To get a complete list of available test plans, run the following:
./tools/catbox-tradefed list plans | grep -i catbox
The following table displays available test plans by type:
Test type | Name |
---|---|
Functional test plans | catbox-functional-notification catbox-functional-setting catbox-functional-dial catbox-functional-system-setting catbox-functional-app-info-setting catbox-functional-network-setting catbox-functional-security-setting catbox-functional-sound-setting catbox-functional-date-time-setting catbox-functional-appgrid catbox-functional-home catbox-functional-lock-screen catbox-functional-navigation-bar catbox-functional-ux-restriction catbox-functional-bluetooth-audio |
Performance test plans | catbox-performance-cold-app-start-up-settings catbox-performance-hot-app-start-up-settings catbox-performance-jank-settings catbox-performance-cold-app-start-up-dialer catbox-performance-hot-app-start-up-dialer catbox-performance-jank-contact-list catbox-performance-jank-notifications catbox-performance-jank-appgrid catbox-performance-cold-app-start-up-mediacenter catbox-performance-hot-app-start-up-mediacenter catbox-performance-create-and-switch-to-new-guest catbox-performance-create-and-switch-to-new-user catbox-performance-create-and-switch-to-precreated-guest catbox-performance-create-and-switch-to-precreated-user catbox-performance-switch-to-existing-user |
CATBox test results
Test results are saved in the android-catbox/results/latest
folder for both
functional and performance test executions.
- For test results, refer to
test_result.xml
. - For failure tests details, see
test_result_failures_suite.html
. - For performance metrics results, see
CatboxPerformanceTests.reportlog.json
in thereport-log-files
directory.
Logs are saved in the path android-catbox/logs/latest
for both functional and
performance test executions.
Some tests such as Dial, Bluetooth Audio, and Contacts list jank tests require additional device setup.
Dial test
The Dial test requires device setup and runtime parameters. To set up the device:
- Connect the mobile device to the Automotive head unit through Bluetooth.
- Make sure the test phone contains a SIM card to run dial tests and is connected to the mobile network.
- Dial tests use default contacts stored on the phone. Use the contacts.vcf to load contacts on the phone.
Replace the
COMPANION_SERIAL
with the Serial ID for the phone and run the following commands:# Contacts can be loaded on the phone by pushing the contacts.vcf file on the phone and importing it in the Contacts application. OR # Importing contacts using adb. # Push contacts VCF file to phone $ adb -s COMPANION_SERIAL push <path-to-downloaded-contacts-vcf-file> /storage/emulated/0/Android/data/com.google.android.contacts/contacts.vcf # Grant Read Permissions $ adb -s COMPANION_SERIAL shell pm grant com.google.android.contacts android.permission.READ_EXTERNAL_STORAGE # Import contacts from the VCF file $ adb -s COMPANION_SERIAL shell am start-activity -W -t 'text/x-vcard' -d file:///storage/emulated/0/Android/data/com.google.android.contacts/contacts.vcf -a android.intent.action.VIEW com.google.android.contacts # Then accept the prompts on the UI to load the contacts.
Runtime parameters
Dial tests use small and large phone numbers for test execution. Dial tests also use the contact name and contact number for search tests. Use the following parameters to provide this information:
Parameters for Large Phone Number:
--module-arg AndroidAutomotiveDialTests:instrumentation-arg:large-phone-number:=< 10 digit phone number>
Parameters for Small Phone Number:
--module-arg AndroidAutomotiveDialTests:instrumentation-arg:small-phone-number:=< 3 digit phone number>
Parameters for Search Contact Name:
--module-arg AndroidAutomotiveDialTests:instrumentation-arg:search-contact-name:=< some contact name stored on the phone >
Parameters for Search Contact Number:
--module-arg AndroidAutomotiveDialTests:instrumentation-arg:search-contact-number:=< contact number of the name provided in the search contact name >
Bluetooth audio test
To set up a device for the Bluetooth audio test:
- Ensure that the YouTube Music app is installed on the phone and you are logged in to an account. If not, install the app and sign-in to an account.
- Pair the phone manually to the automotive device through Bluetooth.
- Once the phone is paired, play a song on the phone and then start executing the test.
Contacts list jank test
To set up a device for the Contacts list jank test:
- Make sure the phone has several contacts so that the test can scroll through the Contacts list to collect jank metrics.
- Pair the phone manually to the automotive device through Bluetooth.
- Once the phone is paired, the contacts can be accessed on the device.