Android 11 Camera Image Test Suite release notes

This page summarizes the changes to Camera Image Test Suite (ITS) in Android 11. The changes fall into the following categories:

• Hardware changes
• First API level MANDATED tests
• Test lighting validated
• Scene name changes
• Test changes and additions
• Increased LIMITED camera testing

Hardware changes

Android 11 introduces several hardware changes to reduce cost and increase availability. These changes fall into the following categories:

• Additional manufacturer
• Unified manufacturing methods
• Increased tablet options
• Reduced tablet opening
• New sensor fusion controller

Additional manufacturer

Rahi Systems is qualified to produce ITS test enclosures in addition to our existing supplier, MYWAY design. Company information for qualified vendors is as follows:

• Rahi Systems Inc.
  48303 Fremont Blvd, Fremont CA 94538, USA
  rahisystems.com/products/android-device-testing-equipment/
  androidpartner@rahisystems.com
  +1-510-319-3802

• MYWAY design
  4F., No. 163, Fu-Ying Road, XinZhuang District, New Taipei City, Taiwan
  twmyway.com
  sales@myway.tw
  +886-2-29089060

Unified manufacturing methods

The rev1 regular field-of-view (RFoV) ITS-in-a-box test enclosure is redesigned to use the manufacturing methods in use by the wide field-of-view (WFoV) box and sensor fusion box test enclosures. The functionality is identical, and for simplicity, the design is referred to as rev1a. The redesign allows manufacturers to stock a single type of plastic to manufacture all test enclosures. Additionally, the tablet mount and light holders are redesigned to handle greater variation in tablets and LED light bars.

To download the latest descriptions and mechanical drawings, see RFoV box (rev1a) and WFoV box (rev2.9).

Increased tablet options

Tablets including the Samsung Galaxy Tab A 10.1 and the Chuwi Hi9 Air 10.1 are added to the list of recommended tablets. It's important the tablet doesn't have pulse width modulation (PWM) to adjust screen brightness to eliminate banding in captured images.

For the latest information on recommended tablets, see Tablet requirements.

Reduced tablet opening

To allow usage of the Galaxy Tab A 10.1, the tablet opening is reduced slightly in height for both the RFoV (rev1a) and WFoV (rev2) test enclosures. The revisions that reflect these changes are rev1a.1 and rev2.9. For these drawings, see RFoV box (rev1a) and WFoV box (rev2.9).

New sensor fusion controller

The hardware for the sensor fusion controller is redesigned to improve manufacturability. The new controller is Arduino based, with a custom routing board shield that mounts on top of the Arduino. Figure 1 shows the shield and figure 2 shows the mechanical drawing for the enclosure. The new controller is powered by a single 5 V supply that powers the motor directly. The electronics are controlled completely through the USB connector. The separate power supply allows for complete isolation between the control electronics and the servo motor. Additionally, a single controller can control up to six servo motors.

Figure 1. Top view of Arduino shield

Figure 2. Enclosure design

Android 11 is backward compatible with existing controllers. To invoke testing with the Arduino-based controller use:

python tools/run_all_tests.py device=# camera=# rot_rig=arduino:1 scenes=sensor_fusion

First API level

In Android 10, ITS tests are designated as MANDATED and NOT_YET_MANDATED. To launch as an Android 10 device, all MANDATED tests must pass. The NOT_YET_MANDATED tests can fail but are tabulated as PASS for CTS verifier reporting. The MANDATED tests requirement also applies to upgraded devices. This requirement for upgraded devices to pass all MANDATED tests caused tests to be delayed in becoming MANDATED tests because older devices must also pass the tests.

In Android 11, MANDATED tests are gated by the first API level flag from the phone properties. For devices upgrading to Android 11, tests run as NOT_YET_MANDATED tests, meaning that a test can fail but be tabulated as PASS in CtsVerifier.apk.

For example:

• In Android 11, the test_channel_saturation test is MANDATED for devices with a first API level greater than 29.
• In Android 10, the test_channel_saturation test is MANDATED for all devices.

Validating scene lighting

In Android 11, the scene lighting is validated by analyzing the brightness in the corners of the scene. All manual scenes are validated for lighting, and tablet-based scenes are validated for RFoV cameras in the RFoV test rig and WFoV cameras in the WFoV test rig. If lighting levels are inadequate, an error is reported and testing fails.

Scene name changes

In Android 10, scene 1 accounts for the majority of tests and a large percentage of the total test time. If any test within scene 1 fails, the entire scene must be rerun. By design, rerunning the entire scene reduces the passage of marginal tests. In Android 11, rerun times are reduced by splitting scene 1 into two scenes, scene1_1 and scene1_2.

The following table shows the test times tabulated for the Pixel 4 rear camera for different scenes. The number of tests is split to equalize test time, not to equalize the number of tests.

Additionally, there is a name cleanup. Scene 2 is split with letters and scene 1 is split with numbers. The nomenclature for the different extensions is:

• Scenes with same chart, but different tests: *_1,2,3
• Scenes with different charts, but same tests: *_a,b,c

Test changes

Tests updated to use first API level

In Android 11, the tests in the following table are updated to use the first API level flag. All of these tests use a first API level of 29 except for the test_tonemap_curve test, which uses a first API level of 30.

Tests with changes

The tests in the following table are updated in Android 11. The changes are described in the Description of changes column.

New tests

The tests in the following table are enabled in Android 11. The tests are summarized in the table and detailed descriptions are provided in the following sections.

scene0/test_vibration_restriction

This test requires no specific scene, but the device under test (DUT) must be placed on or mounted on a hard surface. This includes mounting on the ITS-in-a-box test enclosures.

Asserts

• No vibrations during camera usage

scene2_a/test_jpeg_quality

Method

Different portions of the JPEG file are defined by 2-byte markers. For more information, see JPEG.

The test extracts the quantization matrixes from the JPEG capture. The marker for the quantization matrixes in the JPEG capture is the sequence, [255, 219]. When the marker is found, the next two list items are the size. The JPEG DQT size marker is usually [0, 132] = 256*0+132 = 132, which accounts for the size of the DQT data in the JPEG capture. The embedded data is of the form: [255, 219, 0, 132, 0 (luma marker), 8x8 luma matrix, 1 (chroma marker), 8x8 chroma matrix].

The 0 for the luma matrix marker and 1 for the chroma marker appears consistent for a number of devices including phones that separate the two matrixes into separate DQT sections in the JPEG file. Luma matrixes tend to have a higher variety of values compared to chroma matrixes as the human eye is more sensitive to luma than chroma and JPEG images take this into account.

Sample extracted luma and chroma matrixes are shown below for quality factors of 85 and 25 for the Pixel 4 rear camera capturing scene2_a with the ITS test rig. The matrix values increase (denoting increased compression) substantially for the lower quality setting. These matrixes are only printed with the script if the debug=True flag is applied. Note the larger variation in entries in the luma matrixes compared to the chroma matrixes.

    luma matrix (quality = 85)    chroma matrix (quality = 85)

    [[ 5  3  4  4  4  3  5  4]    [[ 5  5  5  7  6  7 14  8]
     [ 4  4  5  5  5  6  7 12]     [ 8 14 30 20 17 20 30 30]
     [ 8  7  7  7  7 15 11 11]     [30 30 30 30 30 30 30 30]
     [ 9 12 17 15 18 18 17 15]     [30 30 30 30 30 30 30 30]
     [17 17 19 22 28 23 19 20]     [30 30 30 30 30 30 30 30]
     [26 21 17 17 24 33 24 26]     [30 30 30 30 30 30 30 30]
     [29 29 31 31 31 19 23 34]     [30 30 30 30 30 30 30 30]
     [36 34 30 36 28 30 31 30]]     [30 30 30 30 30 30 30 30]]

    luma matrix (quality = 25)            chroma matrix (quality = 25)

    [[ 32  22  24  28  24  20  32  28]    [[ 34  36  36  48  42  48  94  52]
     [ 26  28  36  34  32  38  48  80]     [ 52  94 198 132 112 132 198 198]
     [ 52  48  44  44  48  98  70  74]     [198 198 198 198 198 198 198 198]
     [ 58  80 116 102 122 120 114 102]     [198 198 198 198 198 198 198 198]
     [112 110 128 144 184 156 128 136]     [198 198 198 198 198 198 198 198]
     [174 138 110 112 160 218 162 174]     [198 198 198 198 198 198 198 198]
     [190 196 206 208 206 124 154 226]     [198 198 198 198 198 198 198 198]
     [242 224 200 240 184 202 206 198]]     [198 198 198 198 198 198 198 198]]

Figure 3 shows the average matrix values for the Pixel 4 rear camera versus JPEG quality. As JPEG quality is increased, the level of compression (luma/chroma DQT matrix average) decreases.

Figure 3. Pixel 4 rear camera luma/chroma DQT matrix averages versus JPEG quality

Asserts

• For [25, 45, 65, 86], +20 in quality has 20% reduction quantization matrix averages.
• DQT matrix payloads are square numbers.

Figure 4 shows an example of a phone that fails the test. Note that for very low quality images (jpeg.quality < 50), there is no increase in compression in the quantization matrix.

Figure 4. Failed test example

scene2_d/e test_num_faces

Two new face detection scenes are added to increase the facial diversity of the face detection algorithm checks. With repeated testing of a number of cameras, the most challenging face is expected to the be the leftmost face in scene2_d. In particular, there is both a hat and a beard on the model, something new in the face scenes. The new scenes are shown in figure 5 and 6.

Figure 5. scene2_d

Figure 6. scene2_e

Asserts

• num_faces == 3

scene2_e/test_continuous_picture

Method

The test_continuous_picture test makes use of scene2_e but it can be enabled with any of the faces scenes. In this test, 50 frames of VGA resolution are captured with the capture request first setting android.control.afMode = 4 (CONTINUOUS_PICTURE).

The 3A system is expected to have settled at the end of a 50-frame capture.

Asserts

• 3A is in converged state at end of capture.

scene_change/test_scene_change

Method

A new test is enabled to test if the android.control.afSceneChange flag is asserted with a scene change. The scene change makes use of the tablet displaying a face scene and then turning the tablet on and off to create a scene change. The scene reuses scene2_e but is in a separate scene because of the required tablet control.

Additionally, for manual testing, the scene change can be accomplished by waving your hand in front of the camera.

Figure 7 shows a timing diagram of the test. Timing between the screen turning off and the capture is adjusted based on event results from previous captures.

Figure 7. Timing diagram for test_scene_change

Shift conditions:

• If there is a scene change and afSceneChange == 1, the test returns PASS.
• If there is a scene change and afSceneChange == 0, the scene change shifts 5 frames earlier to give more time for afSceneChange to assert.
• If there is no scene change and afSceneChange == 1, the test returns FAIL.
• If there is no scene change and afSceneChange == 0, the scene change shifts 30 frames earlier to get scene change in capture.

Asserts

• Screen (scene) toggles.
• The afSceneChange flag is in [0, 1].
• If no scene change, 3A converges (functionally identical to test_continuous_picture).
• If afSceneChange == 1, brightness must change in scene.
• PASS within six tries with timing changed based on previous results.

scene6/test_zoom

Method

A new scene is required to test android.control.zoomRatioRange as the established scenes either don't have a feature small enough to be magnified (scenes [1, 2, 4]) or the scene has many objects that aren't easily identified, complicating feature extraction (scene 3).

Figure 8 shows the new scene with a regular array of circles. The array of circles loosens the requirements on DUT/chart centering and allows for a circle always near the center of the captured image. In this scene an array of 9x5 circles with a black border covers the entire tablet. One circle is replaced with a square in the upper right corner to show orientation. The circle sizes have a feature with an area of about 7500 pixels (radius=50pixels) for a 4000x3000 sensor captured with a field of view (FoV) of about 80 degrees.

Figure 8. test_zoom scene

Figure 9. Pixel 4 cam[0] zoom = [1, 3.33, 5.67, 8] images with found circle

Figure 9 shows captured images for the rear camera of a Pixel 4 as the zoom increases from 1 to 8x with four steps. This set of images is captured with no specific care taken in centering except to use the phone testing aperture with two openings to enable testing of both front and back cameras. An offset from center is expected, and is observed as the chart tablet is slightly left of center. Additionally, the chart appears sufficient to test with zoom ratios higher than 8x.

Finding circles

The test includes a find_circle() method using findContours that finds all contours and narrows the contours search down to the desired circles by testing the following:

• Contours must have an area greater than 10 pixels.
• Contours must have NUM_PTS >= 15.
• Contours must have black centers.
• Contours must resemble a circle, that is, their area is close to the pi*r2 area of the contour.

Test range

android.control.zoomRatioRange is divided into 10 steps.

• [1, 7] tests [1, 1.67, 2.33, 3, 3.67, 4.33, 5, 5.67, 6.33, 7]

Zooming is halted if the found circle touches the boundaries of the image. There's a check to make sure a sufficient zoom level is reached in the test (10x).

Asserts

• At least one circle is found at each zoom setting.
• 10x or maximum of android.control.zoomRatioRange is tested.
• Circle radius scales with zoom (RTOL 10% from expected).
• Circle center offset from center scales with zoom (RTOL 10% from expected).
• Sufficient zoom level is reached (2x).

Increased LIMITED camera testing

In Android 11, the tests in the following table test LIMITED cameras. In addition to the new tests, the scene4/test_aspect_ratio_and_crop test is updated to enable testing of LIMITED devices with a first API level of 30 or higher.

Figure 10 shows the Android 11 ITS secret decoder ring. The secret decoder ring shows what test settings individual tests are gated by. The gating is color coded for simplicity in viewing. The main gating items are:

• MANUAL_SENSOR
• READ_3A *requires MANUAL SENSOR
• COMPUTE_TARGET_EXPOSURES *requires MANUAL SENSOR
• PER_FRAME_CONTROL
• RAW
• SENSORS *REALTIME
• MULTI_CAMERA

MANUAL SENSOR, READ_3A, COMPUTE_TARGET_EXPOSURES, and PER_FRAME_CONTROL gate the majority of tests. Additionally, tests that are enabled for LIMITED devices are highlighted in light green.

Figure 10. Android 11 secret decoder ring