Complete Automotive Tests in a Box

Complete Automotive Tests in a Box (CATBox) is an open source package that provides the frameworks and tools required to streamline and execute automotive tests with minimal configuration. CATBox is scalable and compatible with the tools, infrastructure, and tests you use to test and validate your Android Automotive OS (AAOS) implementations. As a result, you can provide high-quality products to the marketplace.

CATBox design and implementation

The CATBox test suite provides the tools, frameworks, and tests needed to execute functional and performance tests on an automotive target device.

The following image illustrates a high level design of a CATBox package.

Figure 1. CATBox package

Tradefed

Tradefed is an open source continuous testing framework used to execute tests on Android devices. For details, see Trade Federation Overview.

Automotive test framework

CATBox is based on an Automotive test framework called Spectatio, which provides APIs for testing various applications on automotive devices. It is built on top of UI Automator, an open source testing framework, that provides a set of APIs to build UI tests that interact with user and system applications.

CATBox test suite downloads

Android 12

Android 12 is the release of the development milestone code-named S. Use the following links to download CATBox test suites for Android 12:

• Android 12 (arm64)
• Android 12 (x86_64)

Android 11

Android 11 is the release of the development milestone code-named R. Use the following links to download CATBox test suites for Android 11:

• Android 11 (arm64)
• Android 11 (x86_64)

Build CATBox

Use the catbox build command once the Android AOSP code is available on the local workstation. To build the CATBox package locally:

1. Select Device Target and run the following command:

   # Select Device Target Based on Device Architecture
   $ lunch <target>
   

2. To build CATBox, run the following command:

   # Build CATBox
   $ make catbox
   

   This command creates the android-catbox.zip file in the /out/host/linux-x86/catbox directory, which can then be used to run tests.

Environment setup

The environment setup required to execute CATBox tests is similar to Setting up CTS. To set up the environment, complete each of the following tasks:

• Install ADB and AAPT.
• Install JDK. See Java Development Kit for Ubuntu.
• Increase memory limit. See Storage requirements.

CATBox setup and execution

Use the Tradefed test framework to run the tests on Android devices. Before CATBox setup, familiarize yourself with Trade Federation Overview.

To set up and run CATBox, see the sections below.

Device setup

Before running a test plan, complete the following steps to set up your device:

1. Ensure that you have installed the recent versions of both the Android Debug Bridge (adb) and the Android Asset Packaging Tool (AAPT), as well as added the location of these tools to the system path of your machine. For details, see ADB and AAPT.

2. Connect at least one device and prepare the Device Under Test (DUT) as follows:

   1. To factory reset the automotive device, click Settings > System > Reset Options > Erase all data.
   2. Wait for the device to automatically restart.

   3. Make sure adb is enabled. To confirm the device is available, run the following command:

      $ adb devices
      

   4. To enable root, run the following command:

      # Enable Root
      $ adb -s DEVICE_SERIAL root
      

      Replace DEVICE_SERIAL with the Serial ID for the automotive device.

   5. Connect to Wi-Fi.

Extract the CATBox package

After completing the device setup, unzip the CATBox package and go to the extracted folder:

# Extract the CATBox Package
$ unzip android-catbox.zip

# Navigate to extracted CATBox Package
$ cd android-catbox

Execute tests in the CATBox

To run CATBox with the test plans in the CATBox package, run the following:

./tools/catbox-tradefed run commandAndExit TEST_PLAN_NAME --serial DEVICE_SERIAL

Replace DEVICE_SERIAL with the Serial ID for the automotive device. If you have multiple devices connected to the host machine, use DEVICE_SERIAL to distinguish them. Replace TEST_PLAN_NAME with the name of the test plan you want to run.

To get a complete list of available test plans, run the following:

./tools/catbox-tradefed list plans | grep -i catbox

The following table displays available test plans by type:

Test type	Name
Functional test plans	catbox-functional-notification catbox-functional-setting catbox-functional-dial catbox-functional-system-setting catbox-functional-app-info-setting catbox-functional-network-setting catbox-functional-security-setting catbox-functional-sound-setting catbox-functional-date-time-setting catbox-functional-appgrid catbox-functional-home catbox-functional-lock-screen catbox-functional-navigation-bar catbox-functional-ux-restriction catbox-functional-bluetooth-audio
Performance test plans	catbox-performance-cold-app-start-up-settings catbox-performance-hot-app-start-up-settings catbox-performance-jank-settings catbox-performance-cold-app-start-up-dialer catbox-performance-hot-app-start-up-dialer catbox-performance-jank-contact-list catbox-performance-jank-notifications catbox-performance-jank-appgrid catbox-performance-cold-app-start-up-mediacenter catbox-performance-hot-app-start-up-mediacenter catbox-performance-create-and-switch-to-new-guest catbox-performance-create-and-switch-to-new-user catbox-performance-create-and-switch-to-precreated-guest catbox-performance-create-and-switch-to-precreated-user catbox-performance-switch-to-existing-user

CATBox test results

Test results are saved in the android-catbox/results/latest folder for both functional and performance test executions.

• For test results, refer to test_result.xml.
• For failure tests details, see test_result_failures_suite.html.
• For performance metrics results, see CatboxPerformanceTests.reportlog.json in the report-log-files directory.

Logs are saved in the path android-catbox/logs/latest for both functional and performance test executions.

Some tests such as Dial, Bluetooth Audio, and Contacts list jank tests require additional device setup.

Dial test

The Dial test requires device setup and runtime parameters. To set up the device:

1. Connect the mobile device to the Automotive head unit through Bluetooth.
2. Make sure the test phone contains a SIM card to run dial tests and is connected to the mobile network.
3. Dial tests use default contacts stored on the phone. Use the contacts.vcf to load contacts on the phone.

4. Replace the COMPANION_SERIAL with the Serial ID for the phone and run the following commands:

   # Contacts can be loaded on the phone by pushing the contacts.vcf file on
   the phone and importing it in the Contacts application.
   
   OR
   
   # Importing contacts using adb.
   
   # Push contacts VCF file to phone
   
   $ adb -s COMPANION_SERIAL push <path-to-downloaded-contacts-vcf-file> /storage/emulated/0/Android/data/com.google.android.contacts/contacts.vcf
   
   # Grant Read Permissions
   
   $ adb -s COMPANION_SERIAL shell pm grant com.google.android.contacts
   android.permission.READ_EXTERNAL_STORAGE
   
   # Import contacts from the VCF file
   
   $ adb -s COMPANION_SERIAL shell am start-activity -W -t 'text/x-vcard' -d file:///storage/emulated/0/Android/data/com.google.android.contacts/contacts.vcf
   -a android.intent.action.VIEW com.google.android.contacts
   
   # Then accept the prompts on the UI to load the contacts.
   

Runtime parameters

Dial tests use small and large phone numbers for test execution. Dial tests also use the contact name and contact number for search tests. Use the following parameters to provide this information:

Parameters for Large Phone Number:
--module-arg AndroidAutomotiveDialTests:instrumentation-arg:large-phone-number:=< 10 digit phone number>

Parameters for Small Phone Number:
--module-arg AndroidAutomotiveDialTests:instrumentation-arg:small-phone-number:=< 3 digit phone number>

Parameters for Search Contact Name:
--module-arg AndroidAutomotiveDialTests:instrumentation-arg:search-contact-name:=< some contact name stored on the phone >

Parameters for Search Contact Number:
--module-arg AndroidAutomotiveDialTests:instrumentation-arg:search-contact-number:=< contact number of the name provided in the search contact name >

Bluetooth audio test

To set up a device for the Bluetooth audio test:

1. Ensure that the YouTube Music app is installed on the phone and you are logged in to an account. If not, install the app and sign-in to an account.
2. Pair the phone manually to the automotive device through Bluetooth.
3. Once the phone is paired, play a song on the phone and then start executing the test.

Contacts list jank test

To set up a device for the Contacts list jank test:

1. Make sure the phone has several contacts so that the test can scroll through the Contacts list to collect jank metrics.
2. Pair the phone manually to the automotive device through Bluetooth.
3. Once the phone is paired, the contacts can be accessed on the device.