Android 15 Camera Image Test Suite Release Notes

The Android 15 release includes a number of changes to Camera ITS.

This page summarizes the Camera ITS changes for Android 15. The changes fall into the following broad categories:

Python and package versions

Android 15 supports the following Python and package versions:

We strongly encourage partners to use package management software to bundle the correct versions together and change versions between Android release development environments.

The following package is newly required for Android 15:

FAIL* tests

Android 15 uses the FAIL* designation for tests results to indicate that a test fails but isn't yet mandated. As such, tests with the FAIL* designation are reported as a PASS to CTS Verifier.

Tablet allowlist

Due to testing failures with tablets using pulse width modulation (PWM) to dim their screens, Android 15 uses a tablet allowlist to check the tablet type and OS version. For the list of tablets in the allowlist, see tablet allowlist.

Jetpack Camera App testing

Android 15 introduces testing with the Jetpack Camera App (JCA) to emulate third-party app usage. The JCA must be installed on the device under test before testing scene_flash.

Check alignment tool

Android 15 adds a check alignment tool, tools/check_alignment.py, to aid in determining chart alignment at setup. The tool uses the scene4 chart to compute the center of the circle on the tablet relative to the center of the captured scene.

Deprecated tests

In Android 15, the following tests are removed due to their test functions being included in other tests.

Scene Test name
0 test_capture_result_dump
0 test_param_sensitivity_burst
1_1 test_3a
1_1 test_ae_af
1_1 test_param_exposure_time
1_2 test_param_sensitivity
3 test_3a_consistency

New and updated scenes

Android 15 introduces a number of new and updated scenes to increase test coverage as well as reduce test time.

Scene Description
feature_combination Sensor fusion box scene to enable splitting feature combination testing to separate scene.
sensor_fusion Chart for the sensor fusion box is updated to include ArUco markers. The chart must fill the back of the sensor fusion box by printing at 17"x17" (43x43 cm).
scene7 Tablet-based scene with slanted edge and color quadrants for 3A consistency across camera transitions.
scene8 Tablet-based scene with color-tinted regions for AE and AWB region metering testing.
scene9 Tablet-based scene with high entropy image to stress JPEG compression.
scene_flash Sensor fusion rig reflective chart scene to group lighting-controlled tests.
scene_low_light Tablet-based scene with grid of squares of varying shades of gray against a black background.
scene_video Tablet-based scene with moving objects for frame-drop testing.

New tests

Android 15 includes the following new tests.

Scene Test name Description
2_c test_default_camera_hdr Verifies that the built-in camera app captures Ultra HDR by default if the device is classified as performance class 15.
4 test_30_60fps_preview_fov_match Verifies that the Preview stream has the same field of view for 30 fps and 60 fps.
6 test_preview_zoom Verifies the zoom ratio of each preview frame matches the corresponding capture metadata.
6 test_session_characteristics_zoom Verifies the zoom ratio range for all supported session configurations.
7 test_multi_camera_switch Verifies that the Preview stream has consistent 3A behavior across wide-ultrawide camera transitions.
8 test_ae_awb_regions Verifies that the Preview stream AE and AWB metering regions work correctly.
9 test_jpeg_high_entropy Verifies that JPEG captures can be properly compressed with complex scenes.
feature_combination test_feature_combination Verifies that all mixes of stream combinations, preview stabilization, target FPS range, 10-bit HDR video, and Ultra HDR are supported.
flash test_flash_strength Verifies that flash strength control in SINGLE is implemented properly.
flash test_torch_strength Verifies that torch strength control in TORCH mode is implemented properly.
low_light test_low_light_boost_extension Verifies that Low Light Boost AE mode is implemented properly.
sensor_fusion test_lens_intrinsic_calibration Verifies that the optical center of the lens intrinsic changes when the lens moves due to optical image stabilization (OIS).
sensor_fusion test_preview_distortion Verifies that distortion is corrected throughout each preview frame taken at various zoom levels.
video test_preview_frame_drop Verifies that Preview streams don't drop frames with highest resolution videos.

Refactored tests

In Android 15, the following tests are refactored to increase test coverage and group tests more logically.

Scene Test name Description
0 test_request_capture_match Renamed from test_read_write for clarity.
1_1 test_burst_capture Moved from scene0 and refactored to include checks for capture brightness and frame drops.
1_1 test_exposure_x_iso Renamed from test_exposure for clarity.
1_2 test_raw_burst_sensitivity Refactored to capture 4 frames per ISO value to reduce frame-to-frame variation.
1_2 test_raw_sensitivity Refactored to capture 4 frames per ISO value to reduce frame-to-frame variation.
1_2 test_yuv_plus_raw Refactored to apply lens shading correction maps to RAW color planes.
3 test_flip_mirror No longer allows rotated images in Android 15.
3 test_imu_drift Moved from scene0/test_gyro_bias, renamed, and refactored to include checks for rotation vector with the preview stream running.
4 test_aspect_ratio_and_crop Added YUV + private stream and YUV + private + YUV stream combinations.
4 test_video_aspect_ratio_and_crop Confirmed recorded video with HLG10 has colorspace BT.2020.
flash test_auto_flash Moved from scene2_a to scene_flash to group lighting controlled tests. Refactored to use JCA.
flash test_led_snapshot Moved from sensor_fusion scene to scene_flash to group lighting controlled tests.
flash test_preview_min_frame_rate Moved from scene2_a to scene_flash to group lighting controlled tests.
low_light test_night_extension Moved from scene2_night to scene_low_light to create consistent test environment for low light boost extension and night extension.
sensor_fusion test_preview_stabilization Refactored to test 0.9x (if available) and 1x zoom ratios for common resolutions only (1920x1080, 1280x720, and 640x480.)