Using CTS Verifier

The Android Compatibility Test Suite Verifier (CTS Verifier) supplements the Compatibility Test Suite (CTS). While CTS checks APIs and functions that can be automated, CTS Verifier provides tests for APIs and functions that cannot be tested on a stationary device without manual input, such as audio quality, touchscreen, accelerometer, camera, etc.

Requirements

Before running CTS Verifier, ensure you have the following equipment:

  • Android device that has verified Android API compatibility by successfully passing the CTS. This will be the device-under-test (DUT).
  • Linux computer with USB 2.0 compatible port. All connections to the DUT will be through this port.
  • Second Android device with a known compatible Bluetooth, Wi-Fi direct, and NFC Host Card Emulation (HCE) implementation.

Setting up

To setup the CTS Verifier testing environment:

  1. On the Linux computer:
  2. Connect the DUT to the Linux computer.
  3. From a terminal on the Linux computer, install CtsVerifier.apk on the DUT.
    adb install -r -g CtsVerifier.apk
    
  4. Ensure the DUT has the system data and time set correctly.

Running

Launch the CTS Verifier application by tapping the CTS icon on the DUT:

CTS Verifier
icon in launcher
Figure 1. CTS Verifier icon

The app displays several test sets available for manual verification:

CTS Verifier
menu of tests
Figure 2. CTS Verifier menu of tests.

Each test contains a set of common elements (Info, Pass, Fail):

Streaming video
quality verifier
Figure 3. Test elements.
  • Info (?). Tap to display test instructions. Also appears automatically the first time a test is opened.
  • Pass (✓). Tap if the DUT meets the test requirements per the Info instructions.
  • Fail (!). Tap if the DUT does not meet the test requirements per the Info instructions.

Some tests, such as the USB accessory mode and camera calibration test, require additional test setup and instructions as detailed in the following sections.

Testing USB accessory mode

The USB Accessory test requires a Linux computer to run the USB desktop machine (host) program.

  1. Connect the DUT to the Linux computer.
  2. On the computer, execute the cts-usb-accessory program from the CTS Verifier package:
    ./cts-usb-accessory
  3. Wait for a popup message to appear on the DUT, then select OK.
    CTS Verifier
usb accessory test
    Figure 4. USB accessory test.
  4. Go to the USB Accessory Test in the CTS Verifier application.
  5. On the computer, review the output from the console. Example output:
    CTS USB Accessory Tester
    Found possible Android device (413c:2106) - attempting to switch to accessory
    mode...
    Failed to read protocol version
    Found Android device in accessory mode (18d1:2d01)...
    [RECV] Message from Android device #0
    [SENT] Message from Android accessory #0
    [RECV] Message from Android device #1
    [SENT] Message from Android accessory #1
    [RECV] Message from Android device #2
    [SENT] Message from Android accessory #2
    [RECV] Message from Android device #3
    [SENT] Message from Android accessory #3
    [RECV] Message from Android device #4
    [SENT] Message from Android accessory #4
    [RECV] Message from Android device #5
    [SENT] Message from Android accessory #5
    [RECV] Message from Android device #6
    [SENT] Message from Android accessory #6
    [RECV] Message from Android device #7
    [SENT] Message from Android accessory #7
    [RECV] Message from Android device #8
    [SENT] Message from Android accessory #8
    [RECV] Message from Android device #9
    [SENT] Message from Android accessory #9
    [RECV] Message from Android device #10
    [SENT] Message from Android accessory #10
    

Calibrating camera field of view

Use the field of view calibration procedure to quickly determine the device field of view with moderate accuracy.

  1. Setup the test environment:
    1. Print the calibration-pattern.pdf target file on 11” x 17” or A3 size paper.
    2. Mount the printed pattern on a rigid backing.
    3. Orient the camera device and the printed target as shown below:
      Camera
printed target
      Figure 5. Camera printed target
  2. Set the target width:
    1. Measure the distance (in centimeters) between the solid lines on the target pattern to account for printing inaccuracies (~38 cm).
    2. Start the calibration application.
    3. Press the setup button and select Marker distance.
    4. Measure and enter the distance to the target pattern (~100 cm).
    5. Press the back button to return to the calibration preview.
  3. Verify the device and target are placed as shown in the figure and the correct distances have been entered into the setup dialog. The preview will display the image with a vertical line overlaid onto it; this line should align with the center line of the target pattern. The transparent grid can be used with the other vertical lines to ensure that the optical axis is orthogonal to the target.
  4. Run the calibration test:
    1. Select an image resolution (using selector at the bottom left), then tap the screen to take a photo. The test enters calibration mode and displays the photo with two vertical lines overlaid onto the image.
    2. Determine accuracy:
      • If the lines align with the vertical lines on the target pattern within a few pixels, the reported field of view for the selected resoultion is accurate.
      • If the lines do not align, the reported field of view is inaccurate. To correct, adjust the slider at the bottom of the screen until the overlay aligns with the target pattern as closely as possible. When the overlay and the target pattern image are aligned, the displayed field of view will be a close approximation to the correct value. The reported field of view should be within +/-1 degree of the calibration value.
    3. Press back button and repeat the calibration test for all image resolutions supported by the DUT.

Exporting results

After all tests complete, you can save the results as a report and download to a computer. Report names are automatically time-stamped based on the DUT system time.

  1. Tap the Save (disk) icon.
    CTS Verifier
Save icon
    Figure 6. CTS Verifier Save icon.
  2. Wait for the popup message to display the path to the saved report (e.g. /sdcard/verifierReports/ctsVerifierReport-date-time.zip), then record the path.
    CTS Verifier path to saved report
    Figure 7. CTS Verifier path to saved report.
  3. Connect the DUT to the Linux computer.
  4. From the Android SDK installation on the Linux computer, download reports from the connected device using adb pull CTSVerifierReportPath.
    • For Android 7.x and later, download all reports using:
      adb pull /sdcard/verifierReports
      
    • For Android 6.0 and earlier, download all reports using:
      adb pull /mnt/sdcard/ctsVerifierReports/
      
  5. To clear Pass/Fail results, select the results in the CTS Verifier app and select Menu > Clear.