Complete Automotive Tests in a Box

Complete Automotive Tests in a Box (CATBox) is an open source package that provides the frameworks and tools required to streamline and execute automotive tests with minimal configuration. CATBox is scalable and compatible with the tools, infrastructure, and tests you use to test and validate your Android Automotive OS (AAOS) implementations. As a result, you can provide high-quality products to the marketplace.

CATBox design and implementation

The CATBox test suite provides the tools, frameworks, and tests needed to execute functional and performance tests on an automotive target device and while using a companion device.

The following image illustrates a high level design of a CATBox package.

CATBox

Figure 1. CATBox package

Tradefed

Tradefed is an open source continuous testing framework used to execute tests on Android devices. For details, see Trade Federation Overview.

Automotive test framework

CATBox is based on an Automotive test framework called Spectatio, which provides APIs for testing various apps on automotive devices. It is built on top of UI Automator, an open source testing framework, that provides a set of APIs to build UI tests that interact with user and system applications.

CATBox test suite downloads

Android 14

Android Automotive 14 is the release of the development milestone code-named U. Use the following links to download CATBox test suites for Android 14:

Android 12

Android 12 is the release of the development milestone code-named S. Use the following links to download CATBox test suites for Android 12:

Android 11

Android 11 is the release of the development milestone code-named R. Use the following links to download CATBox test suites for Android 11:

Build CATBox

Use the catbox build command once the Android AOSP code is available on the local workstation.

To build the CATBox package locally:

  1. Select the Device Target based on the device architecture and run the following command:

    lunch <target>
    
  2. To build CATBox, run the following command:

    make catbox
    

    This command creates the android-catbox.zip file in the /out/host/linux-x86/catbox directory, which can then be used to run tests.

Set up the environment

The environment setup required to execute CATBox tests is similar to Setting up CTS. To set up the environment, complete each of the following tasks:

Set up and run CATBox

Use the Tradefed test framework to run the tests on Android devices. Before you set up CATBox, see the Trade Federation Overview.

To set up and run CATBox, see the sections below.

Set up the device

Before running a test plan, complete the following steps to set up your device:

  1. Ensure that you have installed the recent versions of both the Android Debug Bridge (adb) and the Android Asset Packaging Tool (AAPT), as well as added the location of these tools to the system path of your machine. For details, see ADB and AAPT.

  2. Connect at least one device and prepare the Device Under Test (DUT) as follows:

    1. To factory reset the automotive device, go to Settings > System > Reset Options > Erase all data.

    2. Wait for the device to automatically restart.

    3. Make sure adb is enabled. To confirm the device is available, run the following command:

      adb devices
      
    4. To enable root, run the following command:

      adb -s DEVICE_SERIAL root
      

      Replace DEVICE_SERIAL with the Serial ID for the automotive device.

    5. Connect to Wi-Fi.

Extract the CATBox package

After setting up the device, unzip the CATBox package and go to the extracted folder:

unzip android-catbox.zip
cd android-catbox

Execute tests in CATBox

To run CATBox with the test plans in the CATBox package:

./tools/catbox-tradefed run commandAndExit TEST_PLAN_NAME --serial DEVICE_SERIAL

Replace DEVICE_SERIAL with the Serial ID for the automotive device. If you have multiple devices connected to the host machine, use DEVICE_SERIAL to distinguish them. Replace TEST_PLAN_NAME with the name of the test plan you want to run.

To get the most complete list of available plans, use the command line:

./tools/catbox-tradefed list plans | grep -i catbox

Types of test plans

The following table displays the available functional test plans.

Functional test plans Description
catbox-functional Functional tests.
catbox-functional-app-info-setting App info setting functional tests.
catbox-functional-admin-user-grant-permissions Admin user grant permissions functional tests.
catbox-functional-app-info-setting-ui-elements App info setting functional tests.
catbox-functional-appgrid App grid functional tests.
catbox-functional-base List of functional tests.
catbox-functional-bluetooth-audio Bluetooth audio Functional tests.
catbox-functional-bluetooth-palette Status bar functional tests.
catbox-functional-bluetooth-tests Bluetooth tests using Mobly.
catbox-functional-brightness-palette Status bar functional tests.
catbox-functional-btmoped Bluetooth Moped-based functional tests.
catbox-functional-date-time-setting Date and time setting functional tests.
catbox-functional-dial Dial functional tests.
catbox-functional-display-setting Display setting functional tests.
catbox-functional-driving-optimized-apps UX restriction functional tests.
catbox-functional-enable-developers-option Enable developers option functional tests.
catbox-functional-home Home functional tests.
catbox-functional-lock-screen Lock screen functional tests.
catbox-functional-mediacenter Media Center functional tests.
catbox-functional-microphone-recent-apps Microphone setting functional tests.
catbox-functional-microphone-setting Microphone setting functional tests.
catbox-functional-multiuser Multiuser functional tests.
catbox-functional-multiuser-system-user Multiuser functional tests.
catbox-functional-navigation-bar Navigation bar functional tests.
catbox-functional-network-palette Status bar functional tests.
catbox-functional-network-setting Network setting functional tests.
catbox-functional-notification Notification functional tests.
catbox-functional-privacy-permission-manager Privacy App Permission setting functional tests.
catbox-functional-privacy-setting Microphone setting functional tests.
catbox-functional-privacy-setting-ui-elements Privacy setting functional tests.
catbox-functional-profile-icon Profile Icon Bar functional tests.
catbox-functional-profiles-icon-list Profiles icon list tests.
catbox-functional-security-setting Security setting functional tests.
catbox-functional-setting Setting functional tests.
catbox-functional-settings-location Setting location Tests.
catbox-functional-sound-setting Sound setting functional tests.
catbox-functional-status-bar Status bar functional tests.
catbox-functional-storage-setting Storage setting functional tests.
catbox-functional-system-setting System setting functional tests.
catbox-functional-systemui System UI tests.
catbox-functional-ux-restriction UX restriction functional tests.
catbox-functional-ux-restriction-facet-bar UX restriction functional tests.

The following table lists CATBox performance test plans.

Performance test plans Description
catbox-performance-cold-app-start-up-dialer Cold app start-up performance test for Dialer and Phone.
catbox-performance-cold-app-start-up-mediacenter Cold app start-up performance test for Media Center.
catbox-performance-cold-app-start-up-settings Cold app start-up performance test for settings.
catbox-performance-create-and-switch-to-new-guest Measure latency to switch to a new guest.
catbox-performance-create-and-switch-to-new-user Measure latency to switch to a new non-admin user.
catbox-performance-hot-app-start-up-dialer Hot app start-up performance test for Dialer and Phone.
catbox-performance-hot-app-start-up-mediacenter Hot app start-up performance test for Media Center.
catbox-performance-hot-app-start-up-settings Hot app start-up performance test for Settings.
catbox-performance-jank-appgrid Performance test for Appgrid.
catbox-performance-jank-contact-list Performance test for Contacts.
catbox-performance-jank-media Performance tests for Media.
catbox-performance-jank-media-switch-playback Performance test for media switch playback.
catbox-performance-jank-notifications Performance test for Notifications.
catbox-performance-jank-settings Performance test for Settings.
catbox-performance-switch-to-existing-user Measure latency to switch to a existing user.

Run the multidevice host side test

The execution of multidevice host side tests uses CATBox to run the test in one session. For example, BTDiscoveryTest:

make catbox
./tools/catbox-tradefed run commandAndExit catbox-functional-bluetooth-tests --{phone}serial <serial> --{auto}serial <serial> --mobly-host:mobly-par-file-name BTDiscoveryTest

CATBox installs the apk and then the test is run in the correct Python execution environment.

Review CATBox test results

Results of functional and performance test runs are saved in android-catbox/results/latest.

Result type Location
Test results test_result.xml
Failure tests details test_result_failures_suite.html
Performance metrics results /report-log-files/CatboxPerformanceTests.reportlog.json
Functional and performance test runs /android-catbox/logs/latest