Add new pixel formats to Android

All new pixel formats added to Android must be included in the Android Interface Definition Language (AIDL) and in the Android Hardware Buffer (AHB). The AIDL and the AHB have strict stability and standardization requirements that necessitate a careful process when extending functionality. All new pixel formats must land in AOSP and all the updates must be individually confirmed by AIDL and AHB experts. This process of careful confirmation is an important factor in standardizing any new pixel formats on the platform.

This page outlines the necessary AOSP code changes and the process required for adding new pixel formats on AOSP.

Before adding a new pixel format, download the source and upload patches as outlined in Submitting patches.

Add a new pixel format to AIDL

Adding support for a new pixel format requires changes to both of the PixelFormat.aidl files located within AIDL. See hardware/interfaces/graphics/common/aidl/ for the AIDL source code.

To add a new pixel formal to AIDL, follow these steps:

Append the new pixel format as a new entry to the end of the PixelFormat enum in PixelFormat.aidl by following the existing code convention and setting the hex value for your entry to be one more than the previous entry. Match your code changes to the previous entries. See the following example for the RGBA_8888 pixel format entry:

/**
 * 32-bit format that has 8-bit R, G, B, and A components, in that order,
 * from the lowest memory address to the highest memory address.
 *
 * The component values are unsigned normalized to the range [0, 1], whose
 * interpretation is defined by the dataspace.
 */
RGBA_8888 = 0x1,

The following error message is seen when you build the code after making changes to PixelFormat.aidl:

android_developer:~/android/aosp-main: m
...
###############################################################################
# ERROR: AIDL API change detected                                             #
###############################################################################
Above AIDL file(s) has changed. Run `m android.hardware.graphics.common-update-api` to reflect the changes
to the current version so that it is reviewed by
android-aidl-api-council@google.com
And then you need to change dependency on android.hardware.graphics.common-V(n)-* to android.hardware.graphics.common-V(n+1)-* to use
new APIs.

To clear this error, run the following command, as specified in the error message, to change PixelFormat.aidl in the aidl_api directory:
```
m android.hardware.graphics.common-update-api
```
Running the above command updates the correct file to be able to build normally.

Note: Don't attempt to manually edit this file.

Add a new pixel format to AHB

Adding support for a new pixel format requires changes to hardware_buffer.h and AHardwareBuffer.cpp. See frameworks/native/libs/nativewindow for the AHB source code.

To add a new pixel formal to AHB, follow these steps:

In hardware_buffer.h, append the new pixel format as a new entry to the end of the AHardwareBuffer_Format enum. Follow the existing code conventions.
Using the RGBA_8888 pixel format example, add the new pixel format entry as follows:
```
/**
 * Corresponding formats:
 *   Vulkan: VK_FORMAT_R8G8B8A8_UNORM
 *   OpenGL ES: GL_RGBA8
 */
AHARDWAREBUFFER_FORMAT_R8G8B8A8_UNORM = 1,
```
Note that the new pixel format is given a name in AHB, which must begin with AHARDWAREBUFFER_FORMAT_, followed by the channel abbreviations and bit depths, and ending with the encoding. This enum entry must have the same hex value as that in PixelFormat.aidl.

The pixel format is expected to have one or both of an associated Vulkan or OpenGL ES format. Specify the associated format where appropriate. If no associated format exists, specify N/A.
Add the pixel format to optional testing under CTS, if it has an associated OpenGL ES format. To do this, add the new GL format to AHardwareBufferGLTest.cpp in AHBFormatAsString(int32_t format) with FORMAT_CASE(...) and GL_FORMAT_CASE(...) for the new format, shown as follows:
```
const char* AHBFormatAsString(int32_t format) {
  switch (format) {
      ...
      FORMAT_CASE(R8G8B8A8_UNORM);
      ...
      GL_FORMAT_CASE(GL_RGB8);
  }
  return "";
}
```

Next, add a new test to AHardwareBufferGLTest.cpp, shown as follows:

class RGBA8Test : public AHardwareBufferGLTest {};

// Verify that if we can allocate an RGBA8 AHB we can render to it.
TEST_P(RGBA8Test, Write) {
    AHardwareBuffer_Desc desc = GetParam();
    desc.usage = AHARDWAREBUFFER_USAGE_GPU_FRAMEBUFFER;
    if (!SetUpBuffer(desc)) {
        return;
    }

    ASSERT_NO_FATAL_FAILURE(SetUpFramebuffer(desc.width, desc.height, 0, kBufferAsRenderbuffer));
    ASSERT_NO_FATAL_FAILURE(
        SetUpProgram(kVertexShader, kColorFragmentShader, kPyramidPositions, 0.5f));

    glDrawArrays(GL_TRIANGLES, 0, kPyramidVertexCount);
    ASSERT_EQ(GLenum{GL_NO_ERROR}, glGetError());
}

INSTANTIATE_TEST_CASE_P(
    SingleLayer, RGBA8Test,
    ::testing::Values(
        AHardwareBuffer_Desc{57, 33, 1, AHARDWAREBUFFER_FORMAT_R16G16_UINT, 0, 0, 0, 0}),
    &GetTestName);

Specify at least one set of AHardwareBuffer_Desc values. Add more values if needed.

In AHardwareBuffer.cpp, find the end of the static asserts found within:

// ----------------------------------------------------------------------------
// Validate hardware_buffer.h and PixelFormat.aidl agree
// ----------------------------------------------------------------------------

Append a new static_assert for the new pixel format, using the PixelFormat:: enum and not with the HAL_PIXEL_FORMAT constant. Using the same example for the RGBA_8888 pixel format from Add a new pixel format to AIDL, add the new pixel format entry as follows:

static_assert(static_cast(aidl::android::hardware::graphics::common::PixelFormat::RGBA_8888) ==
  AHARDWAREBUFFER_FORMAT_R8G8B8A8_UNORM,
"HAL and AHardwareBuffer pixel format don't match");

Add the new pixel format to the appropriate tests, by appending the new pixel format to the end of PrintAhbFormat() in AHardwareBufferTest.cpp. Follow the existing code convention, as shown below:
```
void PrintAhbFormat(std::ostream& os, uint64_t format) {
    switch (format) {
        ...
        FORMAT_CASE(R8G8B8A8_UNORM);
        default: os << "unknown"; break;
    }
}
```
Add the new pixel format to the HardwareBuffer SDK in HardwareBuffer.java: by appending a new entry to @IntDef. For example, the entry for the RGBA_8888 format is shown as follows:
```
@Retention(RetentionPolicy.SOURCE)
@IntDef(prefix = { "RGB", "BLOB", "YCBCR_", "D_", "DS_", "S_" }, value = {
  ...
  RGBA_8888,
})
```
If the component values are not unsigned normalized, then indicate the value explicitly in the variable name. For example, the variable name for an unsigned integer 16-bit red channel only format must be R_16UI, and the same format with an additional unsigned integer 16-bit green channel format must be RG_16UI16UI.
Add the new pixel format as a static int in HardwareBuffer.java, by appending a new public member variable at the end of @Format:
```
@Format
...
/** Format: 8 bits each red, green, blue, alpha */
public static final int RGBA_8888 = 0x1;
```
This enum entry must have the same hex value as that from PixelFormat.aidl and hardware_buffer.h. Follow existing conventions.

Attempting to build with these code changes generates a build error:

android_developer:~/android/aosp-main: m
...
******************************
You have tried to change the API from what has been previously approved.

To make these errors go away, you have two choices:
   1. You can add '@hide' javadoc comments (and remove @SystemApi/@TestApi/etc)
      to the new methods, etc. shown in the above diff.

   2. You can update current.txt and/or removed.txt by executing the following command:
         m api-stubs-docs-non-updatable-update-current-api

      To submit the revised current.txt to the main Android repository,
      you will need approval.
******************************
...

To clear this error, run the following command, as specified in the error message, to change current.txt:

m api-stubs-docs-non-updatable-update-current-api

Running the above command updates the correct file to be able to build normally.

Add the new pixel format to the Java tests, by appending the new pixel format to the end of paramsForTestCreateOptionalFormats() in HardwareBufferTest.java, shown as follows:
```
private static Object[] paramsForTestCreateOptionalFormats() {
  return new Integer[]{
      HardwareBuffer.RGBA_8888
  };
```

Add a new pixel format to Window System integration

To use the new pixel format as the format for a framebuffer in a graphics API, add it to the appropriate Window System Integration (WSI) for the relevant graphics API. For an app or system process using the Vulkan API, update the Vulkan Swapchain. For an app or system process using the OpenGL ES API, update the EGL API.

Vulkan WSI changes for new pixel formats

Update the Vulkan WSI as follows:

Add a new case to the GetNativePixelFormat(VkFormat format) function in swapchain.cpp:

android::PixelFormat GetNativePixelFormat(VkFormat format) {
  ...
  switch (format) {
      ...
      case VK_FORMAT_R8G8B8A8_UNORM:
          native_format = PixelFormat::RGBA_8888;
          break;
      ...
      default:
          ALOGV("unsupported swapchain format %d", format);
          break;
  }
  return native_format;
}

Query the Vulkan extension if the pixel format requires a Vulkan extension to function. For instance side extensions, use instance_data, shown as follows:

bool colorspace_ext = instance_data.hook_extensions.test(ProcHook::EXT_swapchain_colorspace);

For device side extensions, use the following:

bool rgba10x6_formats_ext = false;
uint32_t exts_count;
const auto& driver = GetData(pdev).driver;
driver.EnumerateDeviceExtensionProperties(pdev, nullptr, &exts_count,
                                          nullptr);
std::vector props(exts_count);
driver.EnumerateDeviceExtensionProperties(pdev, nullptr, &exts_count,
                                          props.data());
for (uint32_t i = 0; i < exts_count; i++) {
    VkExtensionProperties prop = props[i];
    if (strcmp(prop.extensionName,
               VK_EXT_RGBA10X6_FORMATS_EXTENSION_NAME) == 0) {
        rgba10x6_formats_ext = true;
    }
}

Google handles the infrastructure required to expose an instance or device extension to swapchain.cpp. The initial change list isn't required to have the extensions correctly setup from the Vulkan loader.

Next, enumerate the format and colorspace pairs:

desc.format = AHARDWAREBUFFER_FORMAT_R10G10B10A10_UNORM;
if (AHardwareBuffer_isSupported(&desc) && rgba10x6_formats_ext) {
  all_formats.emplace_back(
      VkSurfaceFormatKHR{VK_FORMAT_R10X6G10X6B10X6A10X6_UNORM_4PACK16,
                         VK_COLOR_SPACE_SRGB_NONLINEAR_KHR});
  if (colorspace_ext) {
    all_formats.emplace_back(
        VkSurfaceFormatKHR{VK_FORMAT_R10X6G10X6B10X6A10X6_UNORM_4PACK16,
                           VK_COLOR_SPACE_PASS_THROUGH_EXT});
    all_formats.emplace_back(
        VkSurfaceFormatKHR{VK_FORMAT_R10X6G10X6B10X6A10X6_UNORM_4PACK16,
                           VK_COLOR_SPACE_DISPLAY_P3_NONLINEAR_EXT});
}

You must have knowledge of the compatible format and colorspace pairs.

Add the new format to dEQP-VK located at external/deqp.
Update Vulkan Conformance Tests in vktApiExternalMemoryTests.cpp and vktExternalMemoryUtil.cpp by inferring the required changes from the existing source or reaching out to your Android support for information.

EGL changes for new pixel formats

Update the EGL as follows:

In the getNativePixelFormat() function, modify the if-else tree to return the AIDL enum for the new pixel format. Using the example for the RGBA_8888 pixel format:

if (a == 0) {
  ...
} else {
  if (componentType == EGL_COLOR_COMPONENT_TYPE_FIXED_EXT) {
      if (colorDepth > 24) {
          ...
      } else {
          *format = PixelFormat::RGBA_8888;
      }
  } else {
    ...
  }
}

To add the new format to dEQP, add a new entry to the androidFormats enum, shown as follows:
```
static const GLenum androidFormats[] =
{
  ...
  GL_RGBA8,
  ...
};
```

Submit your update

Follow For contributors to spin up your change lists and share them with the appropriate team.