Android Camera Image Test Suite (ITS) is part of Android Compatibility Test Suite (CTS) Verifier and includes tests that verify image content. As of CTS 7.0_r8, CTS Verifier supports ITS test automation via Camera ITS-in-a-box; support for manual tests continues to ensure coverage for all Android device form factors.
ITS-in-a-box brings the following benefits:
- Automation. No human intervention is needed during the test.
- Reduced testing time. Parallel testing of front/back camera cuts testing cycle time by 50%.
- Easier troubleshooting. Consistency of test environment leads to fewer setup errors and increases reproducibility.
- Efficiency. Ability to retry for individual Camera/Scene improves test execution efficiency.
ITS-in-a-box consists of a plastic box that is laser cut from computer-aided design (CAD) drawings, a chart tablet, and a device under test (DUT). You can use the wide field of view (WFoV) ITS-in-a-box, which is capable of testing both WFoV (FoV > 90 degrees) and RFoV (FoV < 90 degrees) cameras, or the regular field of view (RFoV) ITS-in-a-box.
To get started with the Camera ITS-in-a-box:
- Purchase or build a wide field of view (WFoV) or regular field of view (RFoV) ITS-in-a-box.
- Configure a tablet with Camera ITS software.
- Run tests.
- Get results from the DUT.
Configuring the tablet
This section provides step-by-step instructions for setting up a tablet for use with the Camera ITS software. These instructions use a Pixel C as an example tablet. For information on tablet requirements and recommendations, see Tablet requirements.
Note: The Camera ITS python scripts
automatically set the following options on the tablet for you:
Settings > Display > Sleep > After 30 minutes of inactivity
Adaptive brightness > OFF
- Charge the tablet and power it on. If prompted to set up an account, skip it (Camera ITS does not require any account paired with the tablet).
- Update the tablet to Android 7.0 or higher. Android 6.x and lower versions do not support Camera ITS.
- Enable developer mode.
- Return to Settings and select Developer options.
- Stay awake
- USB debugging (This allows the host to run the tablet in debug mode. When you connect the tablet to the host for the first time, the tablet prompts you to "Allow USB debugging?" If the tablet does not display the debug prompt, disconnect then reconnect tablet.)
- Automatic system updates
- Verify apps over USB
- Determine DUT and chart IDs by running
$ adb devicesto list available devices. To determine
chart_id, plug and unplug devices and observe the devices that connect and disconnect.
- Perform three test runs to suppress hints and user prompts that can obscure
charts on the tablet screen.
- Position the tablet face-up on a table (do not attach the tablet to the back panel of the box)
- Run the following command:
python tools/run_all_tests.py device=$device_id camera=0 chart=$chart_id scenes=2,3Scenes 2 and 3 require the tablet to display an image, so the tablet prompts you to "Allow Drive to access photos, media, and files on your device?". Clear this prompt (and prevent future prompts) by pressing Allow.
- Run the command again. The tablet prompts you to "Keep a copy of this file?" and suggests Google Drive. Clear this prompt (and prevent future prompts) by pressing the Drive icon then Cancel for upload to drive.
- Finally, run
tools/run_all_tests.pyand confirm that scenes change automatically as script cycles through different scenes. While most tests will fail (as the camera is not pointed at the chart), you can verify the tablet correctly cycles through the scenes without displaying any prompts or other pop-ups on the screen.
Before running the ITS-in-a-box, ensure your test setup includes the following hardware and software:
- One (1) ITS-in-a-box
- One (1) Pixel C for displaying Scenes, S/N: 5811000011
- Two (2) DUTs that use the same build fingerprint and have the CTS Verifier
7.0_8+ application installed. Example DUTs:
- One (1) Pixel NOF26W for the back camera(0) testing, S/N: FA6BM0305016. To
install the CTS Verifier app, unzip android-cts-verifier.zip then run
adb -s FA6BM0305016 install -r android-cts-verifier/CtsVerifier.apk
- One (1) Pixel NOF26W for the front camera(1) testing, S/N: FA6BM0305439. To
install the CTS Verifier app, unzip android-cts-verifier.zip then run
adb -s FA6BM0305439 install -r android-cts-verifier/CtsVerifier.apk
- One (1) Pixel NOF26W for the back camera(0) testing, S/N: FA6BM0305016. To install the CTS Verifier app, unzip android-cts-verifier.zip then run
Running scenes 0-4
To run scenes 0 to 4 on the front and back camera in parallel:
python tools/run_parallel_tests.py device0=FA6BM0305016 device1=FA6BM0305439 chart=5811000011
You can retry scenes for both front and back cameras or a single camera:
- To retry scenes on front and back cameras in parallel:
python tools/run_parallel_tests.py device0=FA6BM0305016 device1=FA6BM0305439 chart=5811000011 scenes=3,4
- To retry scenes on a single camera:
python tools/run_all_tests.py device=FA6BM0305016 chart=5811000011 camera=1 scenes=3,4
Running scene 5
Scene 5 requires special setup with specific lighting (for details, refer to
CameraITS.pdf in CTS Verifier, which you can download at
Compatibility Test Suite Downloads).
You can run scene 5 separately
(outside of the box) to test two devices in parallel.
- To run scene 5 on front and back cameras on two devices in parallel:
python tools/run_parallel_tests.py device0=FA6BM0305016 device1=FA6BM0305439 chart=5811000011 scenes=5
Figure 3. Camera scene 5
- To run scene 5 for front and back cameras on a single device:
python tools/run_all_tests.py device=FA6BM0305016 camera=0 scenes=5
python tools/run_all_tests.py device=FA6BM0305016 camera=1 scenes=5
You can view results during testing and save the completed results as a report.
- View progress of running tests. The command
run_parallel_testsprints results only after Camera-Scene tests have finished, so to view results during test execution you must use Android Device Monitor or
adb logcatto verify progress and/or view screenshots.
Example adb command:
adb -s FA6BM0305016 logcat -v timeExample screenshots command:
adb -s FA6BM0305016 shell screencap -p /sdcard/screencap.png
adb -s FA6BM0305016 pull /sdcard/screencap.png
- View results. To save Camera ITS results as a report:
- Press Pass and save the report:
Figure 4. Camera ITS report
- Pull reports from the device:
adb -s FA6BM0305016 pull /sdcard/verifierReports
- Unzip the report file and view the test_result.xml.
Figure 5. Camera ITS reports
- Press Pass and save the report:
Tablets must have a display size of around 10 inches with a screen resolution
greater than 2000 x 1500 pixels. The
DISPLAY_LEVEL value must be
CameraITS/tools/wake_up_screen.py according to the tablet model.
The table below lists the values for recommended tablets.
Recommended tablets for ITS testing are shown below:
|Asus Zen Pad 3||9.7||2048 x 1536||9.47 x 6.44 x 0.28||192||Android 6.0|
|Huawei Mediapad m5||10.8||2560 x 1600||10.18 x 6.76 x 0.29||192||Android 8.0|
|Pixel C||10.2||2560 x 1800||9.53 x 7.05 x 0.28||96||Android 8.0|
|Samsung S3||9.7||2048 x 1536||10.76 x 6.65 x 0.24||192||Android 7.0|
|Sony Xperia Z4||10.1||2560 x 1600||10 x 6.57 x 0.24||192||Android 7.0|
Frequently asked qustions (FAQ)
Q1: How do I determine which test rigs I need for my device?
RFoV ITS-in-a-box revision 1 tests regular
field-of-view (RFoV) cameras for scene 0 to scene 4 tests in the
CameraITS/tests directory. RFoV is defined as
60° < FoV < 90°.
For larger FoV cameras, the lights might appear in the images or the charts
might cover too small an area in the FoV, affecting test results.
The WFoV ITS-in-a-box
revision 2 tests wide field-of-view cameras
(WFoV) for scene 0 to scene 4 tests in the
CameraITS/tests directory. WFoV is defined as
FoV >= 90°.
It's functionally identical to revision 1, but larger. The revision 2 test rig
can test both RFoV and WFoV cameras in Android 9 and higher.
sensor fusion box tests the camera/gyroscope timing
offset and multi-camera systems frame sync with tests in
scenes=sensor_fusion. A camera/gyroscope timing offset of less
than 1 ms is required for the
REALTIME feature flag and
Multi-camera devices can be tested with a single rig for static ITS tests
and a sensor fusion rig if the camera has the
A set of example configurations is provided in the table below.
|Example||Camera FoVs||REALTIME?||Recommended rigs||Notes|
|1||75°||No||Rev 1||Android 7.0 or higher|
|2||75°||Yes||Rev 1 + sensor fusion||Android 9 or higher|
|3||75° + 95°||Yes||Rev 2 + sensor fusion||Android 9 or higher|
Q2: How do I debug a single test?
Tests can be run individually for debugging purposes, but the results aren't
CtsVerifier.apk unless the entire scene is run. Results
are printed to the local screen and images are saved in the local directory
instead of the generated
/tmp/tmp### directory when
To run an individual scene:
- Load a scene by adding the
python tools/run_all_tests.py device=# camera=# chart=# scenes=#
Press Control+C to halt tests after the scene is logged as loaded to
If the correct scene is already on the screen, wake up the screen:
python tools/wake_up_screen.py screen=#
Run an individual test.
python tests/scene#/test_*.py device=# camera=#
Plots are then generated in the local directory and
stderrare printed to the screen.
To get more information for debugging, add
python tests/scene#/test_*.py device=# camera=# debug=True
Q3: Why do I need to run failing tests as an entire scene instead of rerunning tests individually?
Tests can be run individually for debugging purposes, but the results are not
CtsVerifier.apk unless the entire scene is run.
Camera ITS ensures that third-party apps have a compatible camera interface. Similar to a unit test, each test stresses a single specification in the camera. To catch unreliable behavior, these tests are expected to pass as a group for an entire scene. For example, although a single unreliable test may pass a rerun of an entire scene, it would be difficult for multiple unreliable tests to pass.
As an extreme example, consider the case where there are 10 tests in a scene
that each has a 50% probability of returning
PASS. By running each
test individually, there's a high chance that an operator can get the camera to
pass Camera ITS. However, if the tests are run in the aggregate as a scene,
there's only a 0.1% chance that the scene will pass.
Q4: How do I run a single scene or reorder the run scenes?
tools/run_all_tests.py runs all scenes in order by
default. However, scenes can be run individually or in a specified order and
be reported to
To run an individual scene (for example scene2):
python tools/run_all_tests.py device=# camera=# chart=# scenes=2
To run more than one scene in a specific order:
python tools/run_all_tests.py device=# camera=# chart=# scenes=3,2
Q5: A number of scene 1 tests fail with the tablet setup but pass with a paper chart. What's wrong?
Ensure that the tablet and test environment meet the following specifications.
Make sure that the tablet meets the following specifications:
- Display size (inches): 10"
- Display size (pixels): greater than 2000 x 1400 pixels
For more details, see Tablet requirements.
Tests may not obtain correct results if the tablet display brightness is too low. To increase the brightness on tablets running Android 7.0 and higher, run:
edit tools/wake_up_screen.py DISPLAY_LEVEL=96 → DISPLAY_LEVEL=192
Box lighting level (requires lux meter)
Make sure that the target lux value at tablet opening is between 100 and 300.
If the lux level is too high,
FAIL. If the lux level is too low, multiple tests fail.
Q6: Why do scenes have objects out of FoV with the WFoV ITS-in-a-box revision 2?
The chart might not be scaled properly. Chart scaling works with Android 9 or higher. Ensure that the chart distance input is correct for the version of the test rig.
WFoV ITS-in-a-box revision 2
python tools/run_all_tests.py ... chart=# dist=22
RFoV ITS-in-a-box revision 1 (
dist=30 is the default)
python tools/run_all_tests.py ... chart=#
Q7: How do I debug sensor fusion tests?
Ensure that you're in a
groups | egrep ‘dialout'
Ensure that the sensor fusion controller is connected by determining if Microchip Technology is connected to the USB port.
lsusb … Bus 003 Device 004: ID 04d8:fc73 Microchip Technology, Inc. …
Run the test multiple times to get a distribution of test attempts with the command:
python tools/run_sensor_fusion_box.py device=A camera=0 num_runs=10 rotator=default
Run outputs will be in the
/tmp/tmp###folder created under the
#is the run number. Common reasons for failure are:
- The phone isn't centered properly.
- Not enough features are found in the image (often a FoV or lighting issue).
- The returned
FAILis valid, and the timing offset between the camera and the gyroscope must be corrected.
Q8: What information should I include when reporting a testing bug?
When reporting a testing bug, include the generated files and images for the test.
- If you ran the test through
tools/run_all_tests.py, attach a zipped
/tmp/tmp###directory to the bug.
- If you ran the test by itself, attach all of the screen outputs and generated images to the bug.
You should also include a bug report. After the test in question fails, use the following command to generate a bug report and attach the generated zip file to the bug.
adb -s device_id bugreport