CameraX Vendor Extensions

Device manufacturers can implement the CameraX OEM vendor library to expose device-specific effects to third-party developers through the CameraX extensions interface. CameraX defines the CameraX extensions interface for vendor-implemented classes loaded at runtime. This page describes how to implement the OEM vendor library and enable it on devices.

Before implementing the vendor library, make sure you understand how the CameraX Jetpack support library works. To learn more about CameraX, see CameraX overview. For more information on vendor extensions, see Vendor extensions.

Architecture

Architecture

Figure 1. Vendor extensions architecture diagram

This diagram describes the architecture of the CameraX vendor extensions. Third-party apps are built against the CameraX extensions library (camera-extensions) and use the camera extensions public API (camera-extensions API). The camera-extensions API is defined by CameraX and is updated when a new version of the camera-extensions library is released. The versioning of the camera-extensions public API and camera-extensions library are the same.

The extensions interface (extensions-interface) is defined by CameraX and allows the camera-extensions library to talk to the OEM vendor library (camera-extensions-stub). The OEM library must have an implementation that matches the extensions interface version. A single implementation of the camera-extensions library can support multiple versions of the extensions interface.

Implementing the OEM vendor library

These implementation instructions use the bokeh (portrait) vendor extension as an example, but you can apply these to other extensions such as the HDR, beauty, and night mode extensions. To do this, copy and paste the code used for the bokeh extension and replace the extension name with the desired extension (for example, replacing BokehImageCaptureExtenderImpl with HdrImageCaptureExtenderImpl).

You aren't required to provide an implementation for every effect. An effect without a vendor implementation defaults to the CameraX implementation. If an effect isn't available, the default implementation doesn't enable the effect and reports to the third-party developer that the effect is unavailable.

Version verification

When loading the OEM library, CameraX verifies that the OEM library contains a version that's compatible with the extensions interface version (referred to as extension-version in this document). To determine version compatibility, CameraX checks only the major and minor versions (for example, 1.0) but doesn't check the patch version because that's used only for bug fixes, not interface changes. The required extension-version for specific APIs is noted in this document. APIs without a specified extension-version are compatible with extension-version 1.0.

To verify the version, CameraX queries the ExtensionVersionImpl interface. CameraX then uses the version reported by the OEM library to determine the functionality that can be called.

Initialization

CameraX starts the initialization process when it has determined the version of the extensions interface implemented by the OEM library. The InitializerImpl.init method signals to the OEM library that an app wants to use extensions. No other calls to the OEM library (aside from version checking) are made until OnExtensionsInitializedCallback is called with a success state.

This method must be implemented as of extension-version 1.1. For details, see the source at InitializerImpl.

ExtenderStateListener interface

CameraX provides hooks into several places in its pipeline to allow the OEM library to set relevant SessionParameters and CaptureRequest values. To allow the OEM library to set these values at the specified times, implement the ExtenderStateListener interface. This interface must be implemented as part of any extender, whether it's a preview, image capture, bokeh, or HDR extender.

Image capture

To support the extension for image capture, implement the corresponding ImageCaptureExtender interface (for example, BokehImageCaptureExtender or HdrImageCaptureExtender).

ImageCaptureExtender includes interfaces required for extensions related to image capture.

The CaptureProcessor interface must be implemented for post processing to be done at the app layer. As of extensions interface version 1.1, only the YUV_420_888 input image format must be supported. The CaptureProcessor interface isn't required if processing is done in the camera HAL.

The following diagram illustrates the image capture process flow.

Image capture process flow

Figure 2. Image capture flow diagram

Example: BokehImageCaptureExtenderImpl

To support the bokeh extension for image capture, implement the BokehImageCaptureExtenderImpl class in the androidx.camera.extensions.impl package.

Preview

To support the extension for preview, implement the corresponding PreviewExtender interface (for example, BokehPreviewExtender or HdrPreviewExtender). The PreviewExtender interface includes the interfaces required for preview-related extensions.

For details, see the source at PreviewExtender.

Image processing for the preview extension can be performed in the camera HAL or the app layer. This is determined by the value of ProcessorType, which is returned by PreviewExtenderImpl.

If the PROCESSOR_TYPE_REQUEST_UPDATE_ONLY type is returned, processing is performed in the HAL through CaptureRequest keys. If the PROCESSOR_TYPE_IMAGE_PROCESSOR type is returned, processing is performed in the app layer by the PreviewImageProcessorImpl interface. This interface operates on an Image and TotalCaptureResult pair. As of extensions interface version 1.1, only the YUV_420_888 input image format must be supported.

The following diagram illustrates the process flow for the preview vendor extension.

Preview flow diagram

Figure 3. Preview flow diagram

Example: BokehPreviewExtenderImpl

To support bokeh for preview, implement the BokehPreviewExtenderImpl class in the androidx.camera.extensions.impl package.

Reference implementation

For a reference OEM vendor library implementation, see /platform/frameworks/support/camera/integration-tests/extensionstestlib/. Note that this implementation performs passthroughs without actually implementing the effects.

Setting up the vendor library on a device

The OEM vendor library isn't built into an app but instead is loaded from the device at runtime by CameraX. The <uses-library> tag declares that the androidx.camera.extensions.impl library, which is defined in the AndroidManifest.xml file, is a dependency of CameraX and must be loaded at runtime. This allows third-party apps using vendor extensions to automatically attempt to load the OEM vendor library. The OEM library is marked as optional so apps can run on devices that don't have the library on the device.

CameraX handles this behavior automatically when an app tries to use a vendor extension as long as the device manufacturer places the OEM library on the device so that it can be discovered by the app.

To set up the OEM library on a device, do the following:

  1. Add a permission file, which is required by the <uses-library> tag, using the following format: /etc/permissions/ANY_FILENAME.xml. For example, /etc/permissions/camera_extensions.xml. The files in this directory provide a mapping of the library named in <uses-library> to the actual file path on the device.

  2. Use the example below to add the required information to the file.

    • name must be androidx.camera.extensions.impl as that's the library that CameraX searches for.
    • file is the absolute path of the file that contains the extensions implementation (for example, /system/framework/androidx.camera.extensions.impl.jar).
    <?xml version="1.0" encoding="utf-8"?>
<permissions>
    <library name="androidx.camera.extensions.impl"
             file="OEM_IMPLEMENTED_JAR" />
</permissions>

Validation

To test your implementation of the OEM vendor library, use the example app at androidx-main/camera/integration-tests/extensionstestapp/, which runs through various vendor extensions.

Frequently asked questions (FAQ)

Are there any restrictions on API levels?

Yes. This depends on the Android API feature set that's required by the OEM vendor library implementation. For example, ExtenderStateListener.onPresetSession() uses the SessionConfiguration.setSessionParameters() call to set a baseline set of tags. This call is available only on API level 28 and higher. For details on specific interface methods, see the API reference documentation.