カメラ拡張機能

デバイス メーカーは、OEM ベンダー ライブラリが提供する Camera Extensions インターフェイスを介して、ボケ、ナイト モード、HDR などの拡張機能をサード パーティの開発者に公開できます。開発者は、 Camera2 Extensions APIとCameraX Extensions APIを使用して、OEM ベンダー ライブラリに実装されている拡張機能にアクセスできます。

サポートされている拡張機能のリストについては、Camera2 と CameraX で同じです。CameraX Extensions APIを参照してください。拡張機能を追加する場合は、 Issue Trackerでバグを報告してください。

このページでは、デバイスで OEM ベンダー ライブラリを実装して有効にする方法について説明します。

建築

次の図は、Camera Extensions インターフェイスまたはextensions-interfaceのアーキテクチャを示しています。 建築

図 1. Camera Extensions のアーキテクチャ図

図に示すように、カメラ エクステンションをサポートするには、OEM ベンダー ライブラリによって提供されるextensions-interfaceを実装する必要があります。 OEM ベンダー ライブラリは、 CameraX Extensions APIとCamera2 Extensions API の2 つの API を有効にします。これらは、それぞれ CameraX アプリと Camera2 アプリで使用され、ベンダー拡張機能にアクセスします。

OEM ベンダー ライブラリを実装する

OEM ベンダー ライブラリを実装するには、 camera-extensions-stubファイルをシステム ライブラリ プロジェクトにコピーします。これらのファイルは、Camera Extensions インターフェイスを定義します。

camera-extensions-stubファイルは、次のカテゴリに分類されます。

重要なインターフェイス ファイル (変更しないでください)

  • PreviewExtenderImpl.java
  • ImageCaptureExtenderImpl.java
  • ExtenderStateListener.java
  • ProcessorImpl.java
  • PreviewImageProcessorImpl.java
  • CaptureProcessorImpl.java
  • CaptureStageImpl.java
  • RequestUpdateProcessorImpl.java
  • ProcessResultImpl.java
  • advanced/AdvancedExtenderImpl.java
  • advanced/Camera2OutputConfigImpl.java
  • advanced/Camera2SessionConfigImpl.java
  • advanced/ImageProcessorImpl.java
  • advanced/ImageReaderOutputConfigImpl.java
  • advanced/ImageReferenceImpl.java
  • advanced/MultiResolutionImageReaderOutputConfigImpl.java
  • advanced/OutputSurfaceImpl.java
  • advanced/RequestProcessorImpl.java
  • advanced/SessionProcessorImpl.java
  • advanced/SurfaceOutputConfigImpl.java

必須の実装 (実装を追加してください)

  • ExtensionVersionImpl.java
  • InitializerImpl.java

Bokeh 拡張クラス (Bokeh 拡張がサポートされている場合は実装)

  • BokehImageCaptureExtenderImpl.java
  • BokehPreviewExtenderImpl.java
  • advanced/BokehAdvancedExtenderImpl.java

Night 拡張クラス (Night 拡張がサポートされている場合は実装)

  • NightImageCaptureExtenderImpl.java
  • NightPreviewExtenderImpl.java
  • advanced/NightAdvancedExtenderImpl.java

自動拡張クラス (自動拡張がサポートされている場合は実装)

  • AutoImageCaptureExtenderImpl.java
  • AutoPreviewExtenderImpl.java
  • advanced/AutoAdvancedExtenderImpl.java

HDR 拡張クラス (HDR 拡張がサポートされている場合は実装)

  • HdrImageCaptureExtenderImpl.java
  • HdrPreviewExtenderImpl.java
  • advanced/HdrAdvancedExtenderImpl.java

Face Retouch 拡張クラス (Face Retouch 拡張機能がサポートされている場合は実装します)

  • BeautyImageCaptureExtenderImpl.java
  • BeautyPreviewExtenderImpl.java
  • advanced/BeautyAdvancedExtenderImpl.java

ユーティリティ (オプション、削除可能)

  • advanced/Camera2OutputConfigImplBuilder.java
  • advanced/Camera2SessionConfigImplBuilder.java

すべての拡張機能に実装を提供する必要はありません。拡張機能を実装しない場合は、 isExtensionAvailable()を設定してfalseを返すか、対応する Extender クラスを削除します。 Camera2 および CameraX Extensions API は、拡張機能が利用できないことをアプリに報告します。

Camera2 および CameraX Extensions API がベンダー ライブラリとやり取りして拡張機能を有効にする方法を見ていきましょう。次の図は、例として Night 拡張機能を使用したエンドツーエンドのフローを示しています。

メインフロー

図 2.夜間延長の実装

  1. バージョンの確認:

    Camera2/X はExtensionVersionImpl.checkApiVersion()を呼び出して、OEM が実装したextensions-interfaceバージョンが Camera2/X でサポートされているバージョンと互換性があることを確認します。

  2. ベンダー ライブラリの初期化:

    InitializerImplベンダー ライブラリを初期化するメソッドinit()があります。 Camera2/X は、Extender クラスにアクセスする前に初期化を完了します。

  3. Extender クラスをインスタンス化します。

    拡張機能の Extender クラスをインスタンス化します。エクステンダーには、ベーシック エクステンダーとアドバンスト エクステンダーの 2 種類があります。すべての拡張機能に対して 1 つのエクステンダー タイプを実装する必要があります。詳細については、 Basic Extender と Advanced Extender の比較を参照してください。

    Camera2/X はインスタンスを作成して Extender クラスと対話し、情報を取得して拡張機能を有効にします。特定の拡張機能について、Camera2/X は Extender クラスを複数回インスタンス化できます。そのため、コンストラクターまたはinit()呼び出しで手間のかかる初期化を行わないでください。 Basic Extender でonInit()呼び出されたとき、または Advanced Extender でinitSession()が呼び出されたときなど、カメラ セッションが開始されようとしているときにのみ、重労働を行います。

    Night 拡張機能の場合、次の Extender クラスが Basic Extender タイプ用にインスタンス化されます。

    • NightImageCaptureExtenderImpl.java
    • NightPreviewExtenderImpl.java

    Advanced Extender タイプの場合:

    • NightAdvancedExtenderImpl.java

  4. 拡張機能の可用性を確認します。

    拡張機能を有効にする前に、 isExtensionAvailable() 、Extender インスタンスを介して、指定されたカメラ ID で拡張機能が利用可能かどうかを確認します。

  5. カメラ情報を使用してエクステンダーを初期化します。

    Camera2/X は Extender インスタンスでinit()を呼び出し、カメラ ID とCameraCharacteristicsを渡します。

  6. クエリ情報:

    Extender クラスを呼び出して、サポートされている解像度などの情報を取得し、推定レイテンシを取得し、拡張機能を有効にする準備として Extender から要求キーを取得します。

  7. Extender で拡張機能を有効にします。

    Extender クラスは、クラスを有効にするために必要なすべてのインターフェイスを提供します。キャプチャ リクエスト パラメータの挿入やポスト プロセッサの有効化など、OEM 実装を Camera2 パイプラインにフックするメカニズムを提供します。

    Advanced Extender タイプの場合、Camera2/X はSessionProcessorImplと対話して拡張機能を有効にします。 Camera2/X は、Extender でcreateSessionProcessor()を呼び出して、 SessionProcessorImplインスタンスを取得します。

次のセクションでは、拡張フローについて詳しく説明します。

バージョン検証

実行時に OEM ベンダー ライブラリをデバイスからロードするとき、Camera2/X は、ライブラリがextensions-interfaceバージョンと互換性があるかどうかを確認します。 extensions-interface 、セマンティック バージョニングまたは MAJOR.MINOR.PATCH (1.1.0 または 1.2.0 など) を使用します。ただし、バージョン検証ではメジャー バージョンとマイナー バージョンのみが使用されます。

バージョンを確認するために、Camera2/X はサポートされているextensions-interfaceバージョンでExtensionVersionImpl.checkApiVersion()を呼び出します。次に、Camera2/X は、OEM ライブラリによって報告されたバージョンを使用して、拡張機能を有効にできるかどうか、およびどの機能を呼び出す必要があるかを判断します。

メジャー バージョンの互換性

拡張インターフェイスのメジャー バージョンが Camera2/X とベンダー ライブラリの間で異なる場合、互換性がないと見なされ、拡張機能は無効になります。

下位互換性

メジャー バージョンが同一である限り、Camera2/X は、以前のextensions-interfaceバージョンで構築された OEM ベンダー ライブラリとの下位互換性を保証します。たとえば、Camera2/X がextensions-interface 1.3.0 をサポートしている場合、1.0.0、1.1.0、および 1.2.0 を実装した OEM ベンダー ライブラリは引き続き互換性があります。これは、ベンダー ライブラリの特定のバージョンを実装した後、Camera2/X は、ライブラリが今後のextension-interfaceバージョンと下位互換性があることを確認することも意味します。

前方互換性

新しいextensions-interfaceのベンダー ライブラリとの上位互換性は、OEM に依存します。拡張機能を実装するためにいくつかの機能が必要な場合は、特定のバージョンから拡張機能を有効にすることができます。この場合、Camera2/X ライブラリのバージョンが要件を満たしている場合は、サポートされているextensions-interfaceバージョンを返すことができます。 Camera2/X バージョンがサポートされていない場合は、99.0.0 などの互換性のないバージョンを返して拡張機能を無効にすることができます。

ベンダー ライブラリの初期化

OEM ライブラリによって実装されたextensions-interfaceバージョンを確認した後、Camera2/X は初期化プロセスを開始します。 InitializerImpl.init()メソッドは、アプリが拡張機能を使用しようとしていることを OEM ライブラリに通知します。

Camera2/X は、OEM ベンダー ライブラリがOnExtensionsInitializedCallback.onSuccess()を呼び出して初期化の完了を通知するまで、(バージョン チェックを除いて) OEM ライブラリへの他の呼び出しを行いません。

extensions-interface 1.1.0 の時点でInitializerImplを実装する必要があります。 OEM ベンダー ライブラリがextensions-interface 1.0.0 を実装している場合、Camera2/X はライブラリの初期化手順をスキップします。

Basic Extender と Advanced Extender の比較

extensions-interfaceの実装には、Basic Extender と Advanced Extender の 2 種類があります。 Advanced Extender は、 extensions-interface 1.2.0 以降でサポートされています。

カメラ HAL で画像を処理する拡張機能、または YUV ストリームを処理できるポスト プロセッサを使用する拡張機能の Basic Extender を実装します。

Camera2 ストリーム構成をカスタマイズし、必要に応じてキャプチャ要求を送信する必要がある拡張機能用に Advanced Extender を実装します。

比較については、次の表を参照してください。

ベーシックエクステンダー高度なエクステンダー
ストリーム構成修理済み
プレビュー: PRIVATEまたはYUV_420_888 (プロセッサが存在する場合)
静止キャプチャ: JPEGまたはYUV_420_888 (プロセッサが存在する場合)		 OEMによるカスタマイズ可能。
キャプチャリクエストの送信Camera2/X のみがキャプチャ リクエストを送信できます。これらのリクエストにパラメータを設定できます。プロセッサが画像キャプチャ用に提供されている場合、Camera2/X は複数のキャプチャ要求を送信し、すべての画像とキャプチャ結果をプロセッサに送信できます。 camera2 キャプチャ リクエストを実行し、結果とイメージを取得するために、 RequestProcessorImplインスタンスが提供されます。

Camera2/X は、 SessionProcessorImplでstartRepeatingとstartCapture呼び出して、プレビューの繰り返し要求を開始し、静止キャプチャ シーケンスをそれぞれ開始するよう OEM に通知します。

カメラ パイプラインのフック
  • onPresetSessionセッション パラメータを提供します。
  • onEnableSession CameraCaptureSession設定された直後に単一のリクエストを送信します。
  • onDisableSession CameraCaptureSession閉じられる前に単一のリクエストを送信します。
  • initSessionキャプチャ セッションを作成するためのカスタマイズされた camera2 セッション構成を初期化して返します。
  • onCaptureSessionStart 、 CameraCaptureSession構成された直後に呼び出されます。
  • onCaptureSessionEndは、 CameraCaptureSession閉じられる前に呼び出されます。
に適しカメラ HAL または YUV 画像を処理するプロセッサに実装された拡張機能。
  • 拡張機能の Camera2 ベースの実装があります。
  • RAW ストリームなどのカスタマイズされたストリーム構成が必要です。
  • インタラクティブなキャプチャ シーケンスが必要です。
サポートされている API バージョンCamera2 拡張機能: Android 13 以降
CameraX 拡張機能: camera-extensions 1.1.0 以降		Camera2 拡張機能: Android 12L 以降
CameraX 拡張機能: camera-extensions 1.2.0-alpha03 以降

アプリの流れ

次の表は、3 種類のアプリ フローと、対応する Camera Extensions API 呼び出しを示しています。 Camera2/X はこれらの API を提供しますが、これらのフローをサポートするには、ベンダー ライブラリを適切に実装する必要があります。これについては、後のセクションで詳しく説明します。

Camera2 拡張機能CameraX 拡張機能
クエリ拡張機能の可用性CameraExtensionCharacteristics . getSupportedExtensions ExtensionsManager. isExtensionAvailable
クエリ情報CameraExtensionCharacteristics. getExtensionSupportedSizes CameraExtensionCharacteristics. getEstimatedCaptureLatencyRangeMillis CameraExtensionCharacteristics. getAvailableCaptureRequestKeys CameraExtensionCharacteristics. getAvailableCaptureResultKeys ExtensionsManager. getEstimatedCaptureLatencyRange

CameraX は、ライブラリ内の残りの情報を処理します。

拡張機能を有効にしたプレビューと静止キャプチャCameraDevice. createExtensionSession

cameraExtensionsSession. setRepeatingRequest

cameraExtensionsSession. capture

 val cameraSelector = ExtensionsManager. getExtensionEnabledCameraSelector

bindToLifecycle(lifecycleOwner, cameraSelector, preview, ...)

ベーシックエクステンダー

Basic Extender インターフェイスは、カメラ パイプラインのいくつかの場所へのフックを提供します。各拡張機能の種類には、OEM が実装する必要がある対応するエクステンダー クラスがあります。

次の表に、OEMS が拡張機能ごとに実装する必要があるエクステンダー クラスを示します。

実装する拡張クラス
夜NightPreviewExtenderImpl.java

NightImageCaptureExtenderImpl.java

HDR HdrPreviewExtenderImpl.java

HdrImageCaptureExtenderImpl.java

自動AutoPreviewExtenderImpl.java

AutoImageCaptureExtenderImpl.java

ボケBokehPreviewExtenderImpl.java

BokehImageCaptureExtenderImpl.java

顔レタッチBeautyPreviewExtenderImpl.java

BeautyImageCaptureExtenderImpl.java

次の例では、 PreviewExtenderImplとImageCaptureExtenderImplプレースホルダーとして使用しています。これらを、実装している実際のファイルの名前に置き換えます。

Basic Extender には、次の機能があります。

  • CameraCaptureSession ( onPresetSession ) を構成するときにセッション パラメーターを挿入します。
  • キャプチャ セッションの開始イベントと終了イベントを通知し、返されたパラメーター ( onEnableSession 、 onDisableSession ) を使用して HAL に通知する単一の要求を送信します。
  • リクエストのキャプチャ パラメータを挿入します ( PreviewExtenderImpl.getCaptureStage 、 ImageCaptureExtenderImpl.getCaptureStages )。
  • YUV_420_888ストリームを処理できるプレビューおよび静止キャプチャ用のプロセッサを追加します。

Camera2/X がextensions-interfaceを呼び出して、上記の 3 つのアプリ フローを実現する方法を見てみましょう。

アプリ フロー 1: 拡張機能の可用性を確認する

BasicExtenderAppFlow1

図 3. Basic Extender でのアプリ フロー 1

このフローでは、Camera2/X はinit()を呼び出さずに、 PreviewExtenderImplとImageCaptureExtenderImplの両方のisExtensionAvailable()メソッドを直接呼び出します。拡張機能を有効にするには、両方の Extender クラスがtrueを返す必要があります。

多くの場合、これは、拡張機能を有効にする前に、特定の拡張タイプが特定のカメラ ID でサポートされているかどうかをアプリが確認する最初のステップです。これは、一部の拡張機能が特定のカメラ ID でのみサポートされているためです。

アプリ フロー 2: クエリ情報

BasicExtenderAppFlow2

図 4. Basic Extender でのアプリ フロー 2

拡張機能が利用可能かどうかを判断した後、アプリは拡張機能を有効にする前に次の情報を照会する必要があります。

  • まだキャプチャ レイテンシの範囲: ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange 、現在のシナリオで拡張機能を有効にすることが適切かどうかを評価するために、アプリのキャプチャ レイテンシの範囲を返します。

  • プレビュー サーフェスとキャプチャ サーフェスでサポートされているサイズ: ImageCaptureExtenderImpl.getSupportedResolutionsとPreviewExtenderImpl.getSupportedResolutions 、イメージ フォーマットのリストと、サーフェスのフォーマットとサイズでサポートされているサイズを返します。

  • サポートされているリクエスト キーと結果キー: Camera2/X は、次のメソッドを呼び出して、サポートされているキャプチャ リクエスト キーと結果キーを実装から取得します。

    • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys
    • ImageCaptureExtenderImpl.getAvailableCapturetResultKeys

Camera2/X は、詳細情報を照会する前に、これらの Extender クラスで最初にinit()を常に呼び出します。

アプリ フロー 3: 拡張機能を有効にしてプレビュー/静止画キャプチャ (HAL 実装)

BasicExtenderAppFlow3

図 5. Basic Extender でのアプリ フロー 3

上の図は、プレビューを有効にして、プロセッサなしで拡張機能を使用してキャプチャするメイン フローを示しています。これは、カメラ HAL が拡張機能を処理することを意味します。

このフローでは、Camera2/X は最初にinit()を呼び出し、次にonInitます。これにより、カメラ セッションが指定された拡張子で開始されようとしていることを通知します。 onInit()で手間のかかる初期化を行うことができます。

CameraCaptureSessionを設定するとき、Camera2/X はonPresetSessionを呼び出してセッション パラメータを取得します。キャプチャ セッションが正常に構成された後、Camera2/X は、キャプチャ パラメータを含むCaptureStageImplインスタンスを返すonEnableSessionを呼び出します。 Camera2/X は、HAL に通知するために、これらのキャプチャ パラメータを含む 1 つのリクエストをすぐに送信します。同様に、キャプチャ セッションが閉じられる前に、Camera2/X はonDisableSessionを呼び出し、返されたキャプチャ パラメータを含む単一のリクエストを送信します。

Camera2/X によってトリガーされる繰り返しリクエストPreviewExtenderImpl.getCaptureStage()によって返されるリクエスト パラメータが含まれます。さらに、静止キャプチャ リクエストには、 ImageCaptureExtenderImpl.getCaptureStages()によって返されるパラメータが含まれます。

最後に、Camera2/X は、カメラ セッションが終了した後にonDeInit()を呼び出します。 onDeinit()でリソースを解放できます。

プレビュー プロセッサ

カメラ HAL に加えて、拡張機能をプロセッサに実装することもできます。

以下で説明するように、 PreviewExtenderImpl.getProcessorTypeを実装して、プロセッサの種類を指定します。

  • PROCESSOR_TYPE_NONE :プロセッサーなし。画像はカメラ HAL で処理されます。

  • PROCESSOR_TYPE_REQUEST_UPDATE_ONLY :プロセッサ タイプにより、最新のTotalCaptureResultに基づく新しいキャプチャ リクエスト パラメータで繰り返しリクエストを更新できます。

    PreviewExtenderImpl.getProcessor 、 TotalCaptureResultインスタンスを処理するRequestUpdateProcessorImplインスタンスを返し、 CaptureStageImplインスタンスを返して繰り返し要求を更新する必要があります。 PreviewExtenderImpl.getCaptureStage()も処理の結果を反映し、最新のCaptureStageImplを返す必要があります。

  • PROCESSOR_TYPE_IMAGE_PROCESSOR :このタイプを使用すると、プロセッサを実装してYUV_420_888画像を処理し、出力をPRIVATEサーフェスに書き込むことができます。

    PreviewExtenderImpl.getProcessorでPreviewImageProcessorImplインスタンスを実装して返す必要があります。プロセッサは、 YUV_420_888入力画像の処理を担当します。出力をプレビューのPRIVATE形式に書き込む必要があります。 Camera2/X は、 PRIVATEの代わりにYUV_420_888サーフェスを使用して、プレビュー用にCameraCaptureSessionを構成します。

    フローについては、次の図を参照してください。

PreviewProcessor

図 6. PreviewImageProcessorImplを使用したプレビュー フロー

PreviewImageProcessorImplインターフェイスはProcessImplを拡張し、次の 3 つの重要なメソッドを持ちます。

  • onOutputSurface(Surface surface, int imageFormat)プロセッサの出力サーフェスを設定します。 PreviewImageProcessorImplの場合、 imageFormat PixelFormat.RGBA_8888などのピクセル形式です。

  • onResolutionUpdate(Size size)入力画像のサイズを設定します。

  • onImageFormatUpdate(int imageFormat)入力画像の画像形式を設定します。現在、それはYUV_420_888のみです。

画像キャプチャ プロセッサ

静止キャプチャの場合、 ImageCaptureExtenderImpl.getCaptureProcessorを使用してCaptureProcessorImplインスタンスを返すことにより、プロセッサを実装できます。プロセッサは、キャプチャされたYUV_420_888画像とTotalCaptureResultインスタンスのリストを処理し、出力をYUV_420_888サーフェスに書き込みます。

静止キャプチャ リクエストを送信する前に、プレビューが有効で実行中であると想定しても問題ありません。

以下の図の流れを参照してください。

CaptureProcessor

図 7. CaptureProcessorImplを使用したスチール キャプチャ フロー

  1. Camera2/X は、静止キャプチャにYUV_420_888形式のサーフェスを使用して、キャプチャ セッションを構成します。 Camera2/X は、以下を呼び出してCaptureProcessorImplを準備します。

    • CaptureProcessorImpl.onImageFormatUpdate()とYUV_420_888 。
    • CaptureProcessorImpl.onResolutionUpdate()と入力画像サイズ。
    • 出力YUV_420_888サーフェスを持つCaptureProcessorImpl.onOutputSurface() 。

  2. ImageCaptureExtenderImpl.getCaptureStages CaptureStageImplのリストを返します。ここで、各要素は、Camera2/X によって送信されるキャプチャ パラメータを持つCaptureRequestインスタンスにマップされます。たとえば、3 つのCaptureStageImplインスタンスのリストを返す場合、Camera2/X は、 captureBurst API を使用して、対応するキャプチャ パラメータを含む 3 つのキャプチャ リクエストを送信します。

  3. 受信した画像とTotalCaptureResultインスタンスはまとめてバンドルされ、処理のためにCaptureProcessorImplに送信されます。

  4. CaptureProcessorImpl 、結果の Image ( YUV_420_888形式) をonOutputSurface()呼び出しで指定された出力サーフェスに書き込みます。 Camera2/X は、必要に応じて JPEG 画像に変換します。

キャプチャ リクエストのキーと結果のサポート

カメラのプレビューとキャプチャに加えて、アプリはズーム、フラッシュ パラメータを設定したり、タップしてフォーカスをトリガーしたりできます。これらのパラメーターは、拡張機能の実装と互換性がない可能性があります。

以下のメソッドがextensions-interface 1.3.0 に追加され、実装がサポートするパラメーターを公開できるようになりました。

  • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys()実装でサポートされているキャプチャ リクエスト キーを返します。
  • ImageCaptureExtenderImpl.getAvailableCaptureResultKeys()キャプチャ結果に含まれるキャプチャ結果キーを返します。

カメラ HAL が拡張機能を処理する場合、Camera2/X はCameraCaptureSession.CaptureCallbackでキャプチャ結果を取得します。ただし、プロセッサが実装されている場合、Camera2/X はProcessResultImplでキャプチャ結果を取得し、これがPreviewImageProcessorImplおよびCaptureProcessorImplのprocess()メソッドに渡されます。 ProcessResultImpl介して Camera2/X にキャプチャ結果を報告する責任があります。

例として、以下のCaptureProcessorImplインターフェイスの定義を参照してください。 extensions-interface 1.3.0 以降では、2 番目のprocess()呼び出しが呼び出されます。

Interface CaptureProcessorImpl extends ProcessorImpl {
    // invoked when extensions-interface version < 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results);
    // invoked when extensions-interface version >= 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results,
            ProcessResultImpl resultCallback, Executor executor);
}

ズーム、タップしてフォーカス、フラッシュ、露出補正などの一般的なカメラ操作については、キャプチャ要求とキャプチャ結果の両方で次のキーをサポートすることをお勧めします。

  • ズーム：
    • CaptureRequest#CONTROL_ZOOM_RATIO
    • CaptureRequest#SCALER_CROP_REGION
  • タップしてフォーカス:
    • CaptureRequest#CONTROL_AF_MODE
    • CaptureRequest#CONTROL_AF_TRIGGER
    • CaptureRequest#CONTROL_AF_REGIONS
    • CaptureRequest#CONTROL_AE_REGIONS
    • CaptureRequest#CONTROL_AWB_REGIONS
  • 閃光：
    • CaptureRequest#CONTROL_AE_MODE
    • CaptureRequest#CONTROL_AE_PRECAPTURE_TRIGGER
    • CaptureRequest#FLASH_MODE
  • 露出補正：
    • CaptureRequest#CONTROL_AE_EXPOSURE_COMPENSATION

1.2.0 以前のバージョンを実装する Basic Extender の場合、CameraX Extensions API は上記のすべてのキーを明示的にサポートします。 extensions-interface 1.3.0 の場合、CameraX と Camera2 の両方が返されたリストを尊重し、それに含まれるキーのみをサポートします。たとえば、1.3.0 の実装でCaptureRequest#CONTROL_ZOOM_RATIOとCaptureRequest#SCALER_CROP_REGIONのみを返すことにした場合、それはアプリでズームのみがサポートされ、タップしてフォーカス、フラッシュ、および露出補正が許可されていないことを意味します。

高度なエクステンダー

Advanced Extender は、Camera2 API に基づくベンダー実装の一種です。このエクステンダー タイプは、 extensions-interface 1.2.0 で追加されました。デバイスの製造元によっては、拡張機能がアプリ レイヤーに実装される場合があります。これは、次の要因に依存します。

  • カスタム ストリーム構成: RAW ストリームなどのカスタム ストリームを構成するか、異なる物理カメラ ID に対して複数のストリームを用意します。

  • Camera2 リクエストを送信する機能:以前のリクエストの結果に基づいてパラメーターを使用してキャプチャ リクエストを送信できる複雑な対話ロジックをサポートします。

Advanced Extender はラッパーまたは中間レイヤーを提供するため、ストリーム構成をカスタマイズして、オンデマンドでキャプチャー要求を送信できます。

実装するファイル

Advanced Extender 実装に切り替えるには、 ExtensionVersionImplのisAdvancedExtenderImplemented()メソッドがtrueを返す必要があります。拡張機能の種類ごとに、OEM は対応するエクステンダー クラスを実装する必要があります。 Advanced Extender の実装ファイルは、高度なパッケージに含まれています。

実装する拡張クラス
夜advanced/NightAdvancedExtenderImpl.java
HDR advanced/HdrAdvancedExtenderImpl.java
自動advanced/AutoAdvancedExtenderImpl.java
ボケadvanced/BokehAdvancedExtenderImpl.java
顔レタッチadvanced/BeautyAdvancedExtenderImpl.java

次の例では、 AdvancedExtenderImplプレースホルダーとして使用しています。実装している拡張機能のエクステンダー ファイルの名前に置き換えます。

Camera2/X がextensions-interfaceを呼び出して 3 つのアプリ フローを実現する方法を見てみましょう。

アプリ フロー 1: 拡張機能の可用性を確認する

AdvancedAppFlow1

図 8. Advanced Extender でのアプリ フロー 1

最初に、アプリは指定された拡張機能がサポートされているかどうかを確認します。

アプリ フロー 2: クエリ情報

AdvancedAppFlow2

図 9. Advanced Extender でのアプリ フロー 2

AdvancedExtenderImpl.init()を呼び出した後、アプリはAdvancedExtenderImplに関する次の情報を照会できます。

  • 推定静止キャプチャ レイテンシ: AdvancedExtenderImpl.getEstimatedCaptureLatencyRange()現在のシナリオで拡張機能を有効にすることが適切かどうかを評価するために、アプリのキャプチャ レイテンシの範囲を返します。

  • プレビューと静止キャプチャでサポートされている解像度:

    • AdvancedExtenderImpl.getSupportedPreviewOutputResolutions()プレビュー サーフェスの形式とサイズでサポートされているサイズ リストへの画像形式のマップを返します。 OEM は、少なくともPRIVATE形式をサポートする必要があります。

    • AdvancedExtenderImpl.getSupportedCaptureOutputResolutions()静止キャプチャ サーフェスのサポートされている形式とサイズを返します。 OEM は、 JPEGとYUV_420_888形式の出力の両方をサポートする必要があります。

    • AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions()画像分析用の追加のYUV_420_888ストリームでサポートされているサイズを返します。画像解析 YUV サーフェスがサポートされていない場合、 getSupportedYuvAnalysisResolutions() nullまたは空のリストを返す必要があります。

  • 利用可能なキャプチャ リクエスト キー/結果( extensions-interface 1.3.0 で追加): Camera2/X は、次のメソッドを呼び出して、サポートされているキャプチャ リクエスト キーと結果キーを実装から取得します。

    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys
    • AdvancedExtenderImpl.getAvailableCaptureResultKeys

詳細については、 「キャプチャ リクエストのキーと結果のサポート」を参照してください。

アプリ フロー 3: 拡張機能を有効にしたプレビュー/静止キャプチャ

AdvancedAppFlow3

図 10. Advanced Extender でのアプリ フロー 3

上の図は、プレビューを開始し、アドバンスト エクステンダー タイプのスチル キャプチャを開始するためのメイン フローを示しています。各ステップを見ていきましょう。

  1. SessionProcessorImplインスタンス

    Advanced Extender のコア実装はSessionProcessorImplにあります。これは、カスタマイズされたセッション構成を提供し、キャプチャ要求を送信してプレビューを開始し、要求をキャプチャします。 AdvancedExtenderImpl.createSessionProcessor()が呼び出されて、 SessionProcessorImplインスタンスが返されます。

  2. initSession

    SessionProcessorImpl.initSession()拡張機能のセッションを初期化します。ここでリソースを割り当て、 CameraCaptureSessionを準備するためのセッション構成を返します。

    入力パラメーターについては、Camera2/X は、プレビュー、静止キャプチャ、およびオプションの YUV 画像分析の出力サーフェス構成を指定します。この出力サーフェス構成 ( OutputSurfaceImpl ) には、 AdvancedExtenderImplの次のメソッドによって取得されるサーフェス、サイズ、および画像形式が含まれています。

    • getSupportedPreviewOutputResolutions()
    • getSupportedCaptureOutputResolutions()
    • getSupportedYuvAnalysisResolutions()

    Camera2SessionConfigImplインスタンスのリストと、 Camera2OutputConfigImplの構成に使用されるセッション パラメータで構成されるCameraCaptureSessionインスタンスを返す必要があります。 Camera2/X によって渡された出力サーフェスに正しいカメラ イメージを出力する必要があります。出力を有効にするためのいくつかのオプションを次に示します。

    • カメラ HAL での処理: SurfaceOutputConfigImpl実装を使用して、出力サーフェスをCameraCaptureSessionに直接追加できます。これにより、提供された出力サーフェスがカメラ パイプラインに構成され、カメラ HAL が画像を処理できるようになります。

    • 中間ImageReaderサーフェス (RAW、YUV など) の処理: ImageReaderOutputConfigImplインスタンスを使用して中間ImageReaderサーフェスをCameraCaptureSessionに追加します。

      中間イメージを処理し、結果イメージを出力サーフェスに書き込む必要があります。

    • Camera2 サーフェス共有を使用する: Camera2OutputConfigImpl インスタンスを別のCamera2OutputConfigImplインスタンスのCamera2OutputConfigImpl getSurfaceSharingOutputConfigs()メソッドに追加することにより、別のサーフェスとのサーフェス共有を使用します。サーフェスの形式とサイズは同一でなければなりません。

    SurfaceOutputConfigImplおよびImageReaderOutputConfigImplを含むすべてのCamera2OutputConfigImplには、一意の ID ( getId() ) が必要です。これは、ターゲット サーフェスを指定し、 ImageReaderOutputConfigImplから画像を取得するために使用されます。

  3. onCaptureSessionStartおよびRequestProcessorImpl

    CameraCaptureSessionが開始され、Camera フレームワークがonConfigured()を呼び出すと、Camera2/X は Camera2 リクエスト ラッパーRequestProcessImplを使用してSessionProcessorImpl.onCaptureSessionStart()を呼び出します。 Camera2/X はRequestProcessImplを実装しています。これにより、キャプチャ リクエストを実行し、 ImageReaderOutputConfigImpl使用されている場合は画像を取得できます。

    RequestProcessImpl API は、リクエストの実行に関して Camera2 CameraCaptureSession API に似ています。違いは次のとおりです。

    • ターゲット サーフェスは、 Camera2OutputConfigImplインスタンスの ID によって指定されます。
    • ImageReaderの画像を取得する機能。

    指定したCamera2OutputConfigImpl ID でRequestProcessorImpl.setImageProcessor()を呼び出して、イメージを受信するImageProcessorImplインスタンスを登録できます。

    Camera2/X がSessionProcessorImpl.onCaptureSessionEnd()を呼び出した後、 RequestProcessImplインスタンスは無効になります。

  4. プレビューを開始して写真を撮る

    Advanced Extender 実装では、 RequestProcessorImplインターフェースを介してキャプチャー要求を送信できます。 Camera2/X は、それぞれSessionProcessorImpl#startRepeatingとSessionProcessorImpl#startCaptureを呼び出して、プレビューまたは静止キャプチャ シーケンスの繰り返し要求を開始するように通知します。これらのプレビュー リクエストと静止キャプチャ リクエストを満たすには、キャプチャ リクエストを送信する必要があります。

    Camera2/X はSessionProcessorImpl#setParametersを介してキャプチャ リクエスト パラメータも設定します。繰り返し要求と単一要求の両方で、これらの要求パラメーターを設定する必要があります (パラメーターがサポートされている場合)。

    少なくともCaptureRequest.JPEG_ORIENTATIONとCaptureRequest.JPEG_QUALITYをサポートする必要があります。 extensions-interface 1.3.0 は、次のメソッドによって公開されるリクエスト キーと結果キーをサポートします。

    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys()
    • AdvancedExtenderImpl.getAvailableCaptureResultKeys()

    開発者がgetAvailableCaptureRequestKeysリストにキーを設定する場合、パラメーターを有効にして、キャプチャ結果にgetAvailableCaptureResultKeysリストのキーが含まれるようにする必要があります。

  5. startTrigger

    SessionProcessorImpl.startTrigger() CaptureRequest.CONTROL_AF_TRIGGERやCaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGERなどのトリガーを開始するために呼び出されます。 AdvancedExtenderImpl.getAvailableCaptureRequestKeys()で通知されなかったキャプチャ リクエスト キーは無視できます。

    startTrigger() extensions-interface 1.3.0 以降でサポートされています。これにより、アプリは拡張機能を使用してタップしてフォーカスとフラッシュを実装できます。

  6. 掃除

    キャプチャ セッションを終了すると、 CameraCaptureSessionを閉じる前にSessionProcessorImpl.onCaptureSessionEnd()が呼び出されます。キャプチャ セッションが閉じた後、 deInitSession()クリーンアップを実行します。

プレビュー、静止画キャプチャ、画像解析をサポート

プレビューと静止キャプチャの両方のユース ケースに拡張機能を適用する必要があります。ただし、待ち時間が長すぎてプレビューをスムーズに表示できない場合は、拡張機能を静止キャプチャにのみ適用できます。

Basic Extender タイプの場合、プレビュー用に拡張機能を有効にするかどうかに関係なく、特定の拡張機能に対してImageCaptureExtenderImplとPreviewExtenderImplの両方を実装する必要があります。 Often, an app also uses a YUV stream to analyze the image content such as finding QR codes or text. To better support this use case , you should support the stream combination of preview, still capture, and a YUV_420_888 stream for configuring CameraCaptureSession . This means that if you implement a processor, then you have to support the stream combination of three YUV_420_888 streams.

For Advanced Extender, Camera2/X passes three output surfaces to the SessionProcessorImpl.initSession() call. These output surfaces are for preview , still capture, and image analysis, respectively. You must ensure that preview and still capture output surfaces show the valid output. However, for the image analysis output surface, ensure it's working only when it's non-null. If your implementation can't support the image analysis stream, you can return an empty list in AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() . This ensures the image analysis output surface is always null in SessionProcessorImpl.initSession() .

Support video capture

The current Camera Extension architecture supports only the preview and still capture use cases. We don't support enabling the extension on the MediaCodec or MediaRecorder surfaces for recording the video. However, it's possible for apps to record the preview output.

Supporting MediaCodec and MediaRecorder surfaces is under investigation.

Extensions interface version history

The following table shows the Camera Extension interface version history. You should always implement the vendor library with the latest version.

Version Added features
1.0.0
  • Version verification
    • ExtensionVersionImpl
  • Basic Extender
    • PreviewExtenderImpl
    • ImageCaptureExtenderImpl
    • Processor
      • PreviewImageProcessorImpl
      • CaptureProcessorImpl
      • RequestUpdateProcessorImpl
1.1.0
  • Library initialization
    • InitializerImpl
  • Expose supported resolutions
    • PreviewExtenderImpl.getSupportedResolutions
    • ImageCaptureExtenderImpl.getSupportedResolutions
1.2.0
  • AdvancedExtender
    • AdvancedExtenderImpl
    • SessionProcessorImpl
  • Get estimated capture latency
    • ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange
1.3.0
  • Expose supported capture request keys/results keys
    • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys
    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys
    • New process() call that takes ProcessResultImpl in PreviewImageProcessorImpl and CaptureProcessorImpl
    • Support trigger type request
      • AdvancedExtenderImpl.startTrigger

Reference implementation

The following reference OEM vendor library implementations are available in frameworks/ex .

  • advancedSample : A basic implementation of Advanced Extender.

  • sample : A basic implementation of Basic Extender.

  • service_based_sample : An implementation that demonstrates how to host Camera Extensions in a Service . This implementation contains the following components:

    • oem_library : A Camera Extensions OEM library for Camera2 and CameraX Extensions APIs that implements Extensions-Interface . This acts as a passthrough that forwards calls from Extensions-Interface to the service. This library also provides AIDL files and wrapper classes to communicate with the service.

      Advanced Extender is enabled by default. To enable the Basic Extender, change ExtensionsVersionImpl#isAdvancedExtenderImplemented to return false .

    • extensions_service : A sample implementation of the Extensions Service. Add your implementation here. The interface to implement in the service is similar to the Extensions-Interface . For example, implementing the IAdvancedExtenderImpl.Stub performs the same operations as AdvancedExtenderImpl . ImageWrapper and TotalCaptureResultWrapper are required to make Image and TotalCaptureResult parcelable.

Set up the vendor library on a device

The OEM vendor library isn't built into an app; it's loaded from the device at runtime by Camera2/X. In CameraX, the <uses-library> tag declares that the androidx.camera.extensions.impl library, which is defined in the AndroidManifest.xml file of the camera-extensions library, is a dependency of CameraX and must be loaded at runtime. In Camera2, the framework loads an extensions service that also declares that the <uses-library> loads the same androidx.camera.extensions.impl library at runtime.

This allows third-party apps using extensions to automatically load the OEM vendor library. The OEM library is marked as optional so apps can run on devices that don't have the library on the device. Camera2/X handles this behavior automatically when an app tries to use a camera extension as long as the device manufacturer places the OEM library on the device so that it can be discovered by the app.

To set up the OEM library on a device, do the following:

  1. Add a permission file, which is required by the <uses-library> tag, using the following format: /etc/permissions/ ANY_FILENAME .xml . For example, /etc/permissions/camera_extensions.xml . The files in this directory provide a mapping of the library named in <uses-library> to the actual file path on the device.

  2. Use the example below to add the required information to the file.

    • name must be androidx.camera.extensions.impl as that's the library that CameraX searches for.
    • file is the absolute path of the file that contains the extensions implementation (for example, /system/framework/androidx.camera.extensions.impl.jar ).
    <?xml version="1.0" encoding="utf-8"?>
<permissions>
    <library name="androidx.camera.extensions.impl"
             file="OEM_IMPLEMENTED_JAR" />
</permissions>

In Android 12 or higher, devices supporting CameraX extensions must have the ro.camerax.extensions.enabled property set to true , which allows for querying whether a device supports extensions. To do this, add the following line in the device make file:

PRODUCT_VENDOR_PROPERTIES += \
    ro.camerax.extensions.enabled=true \

Validation

To test your implementation of the OEM vendor library during the development stage, use the example app at androidx-main/camera/integration-tests/extensionstestapp/ , which runs through various vendor extensions.

After you complete your implementation, use the Camera Extensions Validation Tool to run automated and manual tests to verify that the vendor library is implemented correctly.

Extended scene mode versus Camera Extensions

For the bokeh extension, in addition to exposing it using Camera Extensions, you can expose the extension using the extended scene mode, which is enabled through the CONTROL_EXTENDED_SCENE_MODE key. For more implementation details, see Camera Bokeh .

Extended scene mode has fewer restrictions compared to Camera Extensions for camera2 apps. For example, you can enable extended scene mode in a regular CameraCaptureSession instance that supports flexible stream combinations and capture request parameters. In contrast, camera extensions support only a fixed set of stream types and have limited support for capture request parameters.

A downside of extended scene mode is that you can only implement it in the camera HAL, which means that it must be verified to work across all orthogonal controls available to app developers.

We recommend exposing bokeh using both the extended scene mode and Camera Extensions because apps might prefer to use a particular API to enable bokeh. We recommend first using the extended scene mode because this is the most flexible way for apps to enable the bokeh extension. Then you can implement the camera extensions interface based on the extended scene mode. If implementing bokeh in the camera HAL is difficult, for example, because it requires a post processor running in the app layer to process images, we recommend implementing the bokeh extension using the Camera Extensions interface.

Frequently asked questions (FAQs)

Are there any restrictions on API levels?

Yes. This depends on the Android API feature set that's required by the OEM vendor library implementation. For example, ExtenderStateListener.onPresetSession() uses the SessionConfiguration.setSessionParameters() call to set a baseline set of tags. This call is available only on API level 28 and higher. For details on specific interface methods, see the API reference documentation .

,

Device manufacturers can expose extensions such as bokeh, night mode, and HDR to third-party developers through the Camera Extensions interface provided by the OEM vendor library. Developers can use the Camera2 Extensions API and the CameraX Extensions API to access the extensions implemented in the OEM vendor library.

For a list of supported extensions, which is the same across Camera2 and CameraX, see CameraX Extensions API . If you want to add an extension, file a bug with the Issue Tracker .

This page describes how to implement and enable the OEM vendor library on devices.

Architecture

The following diagram describes the architecture of the Camera Extensions interface or extensions-interface : Architecture

Figure 1. Camera Extensions architecture diagram

As shown in the diagram, to support Camera Extensions, you need to implement the extensions-interface provided by the OEM vendor library. Your OEM vendor library enables two APIs: CameraX Extensions API and Camera2 Extensions API , which are used by CameraX and Camera2 apps, respectively, to access vendor extensions.

Implement the OEM vendor library

To implement the OEM vendor library, copy the camera-extensions-stub files into a system library project. These files define the Camera Extensions interface.

The camera-extensions-stub files are divided into the following categories:

Essential interface files (don't modify)

  • PreviewExtenderImpl.java
  • ImageCaptureExtenderImpl.java
  • ExtenderStateListener.java
  • ProcessorImpl.java
  • PreviewImageProcessorImpl.java
  • CaptureProcessorImpl.java
  • CaptureStageImpl.java
  • RequestUpdateProcessorImpl.java
  • ProcessResultImpl.java
  • advanced/AdvancedExtenderImpl.java
  • advanced/Camera2OutputConfigImpl.java
  • advanced/Camera2SessionConfigImpl.java
  • advanced/ImageProcessorImpl.java
  • advanced/ImageReaderOutputConfigImpl.java
  • advanced/ImageReferenceImpl.java
  • advanced/MultiResolutionImageReaderOutputConfigImpl.java
  • advanced/OutputSurfaceImpl.java
  • advanced/RequestProcessorImpl.java
  • advanced/SessionProcessorImpl.java
  • advanced/SurfaceOutputConfigImpl.java

Mandatory implementations (add your implementation)

  • ExtensionVersionImpl.java
  • InitializerImpl.java

Bokeh extender classes (implement it if Bokeh extension is supported)

  • BokehImageCaptureExtenderImpl.java
  • BokehPreviewExtenderImpl.java
  • advanced/BokehAdvancedExtenderImpl.java

Night extender classes (implement it if Night extension is supported)

  • NightImageCaptureExtenderImpl.java
  • NightPreviewExtenderImpl.java
  • advanced/NightAdvancedExtenderImpl.java

Auto extender classes (implement it if Auto extension is supported)

  • AutoImageCaptureExtenderImpl.java
  • AutoPreviewExtenderImpl.java
  • advanced/AutoAdvancedExtenderImpl.java

HDR extender classes (implement it if HDR extension is supported)

  • HdrImageCaptureExtenderImpl.java
  • HdrPreviewExtenderImpl.java
  • advanced/HdrAdvancedExtenderImpl.java

Face Retouch extender classes (implement it if Face Retouch extension is supported)

  • BeautyImageCaptureExtenderImpl.java
  • BeautyPreviewExtenderImpl.java
  • advanced/BeautyAdvancedExtenderImpl.java

Utilities (optional, can be deleted)

  • advanced/Camera2OutputConfigImplBuilder.java
  • advanced/Camera2SessionConfigImplBuilder.java

You aren't required to provide an implementation for every extension. If you don't implement an extension, set isExtensionAvailable() to return false or remove the corresponding Extender classes. The Camera2 and CameraX Extensions APIs report to the app that the extension is unavailable.

Let's walk through how the Camera2 and CameraX Extensions APIs interact with the vendor library to enable an extension. The following diagram illustrates the end-to-end flow using the Night extension as an example:

Mainflow

Figure 2. Night extension implementation

  1. Version verification:

    Camera2/X calls ExtensionVersionImpl.checkApiVersion() to ensure that the OEM-implemented extensions-interface version is compatible with Camera2/X supported versions.

  2. Vendor library initialization:

    InitializerImpl has a method init() that initializes the vendor library. Camera2/X completes the initialization before accessing the Extender classes.

  3. Instantiate Extender classes:

    Instantiates the Extender classes for the extension. There are two Extender types: Basic Extender and Advanced Extender. You must implement one Extender type for all Extensions. For more information, see Basic Extender versus Advanced Extender .

    Camera2/X instantiates and interacts with the Extender classes to retrieve information and enable the extension. For a given extension, Camera2/X can instantiate the Extender classes multiple times. As a result, don't do heavy-lifting initialization in the constructor or the init() call. Do the heavy lifting only when the camera session is about to start, such as when onInit() is called in Basic Extender or initSession() is called in Advanced Extender.

    For the Night extension, the following Extender classes are instantiated for the Basic Extender type:

    • NightImageCaptureExtenderImpl.java
    • NightPreviewExtenderImpl.java

    And for the Advanced Extender type:

    • NightAdvancedExtenderImpl.java

  4. Check extension availability:

    Before enabling the extension, isExtensionAvailable() checks if the extension is available on the specified camera ID through the Extender instance.

  5. Initialize the Extender with camera information:

    Camera2/X calls init() on the Extender instance and passes it the camera ID and CameraCharacteristics .

  6. Query information:

    Invokes the Extender class to retrieve information such as supported resolutions, still capture estimated latency, and capture request keys from the Extender in preparation for enabling the extension.

  7. Enable extension on the Extender:

    The Extender class provides all the interfaces needed to enable the class. It offers a mechanism to hook OEM implementation into the Camera2 pipeline such as injecting capture request parameters or enabling a post processor.

    For the Advanced Extender type, Camera2/X interacts with SessionProcessorImpl to enable the extension. Camera2/X retrieves the SessionProcessorImpl instance by calling createSessionProcessor() on the Extender.

The following sections describe the extension flow in greater detail.

Version verification

When loading the OEM vendor library from the device at runtime, Camera2/X verifies if the library is compatible with the extensions-interface version. The extensions-interface uses semantic versioning, or MAJOR.MINOR.PATCH, for example, 1.1.0 or 1.2.0. However, only the major and minor versions are used during the version verification.

To verify the version, Camera2/X calls ExtensionVersionImpl.checkApiVersion() with the supported extensions-interface version. Camera2/X then uses the version reported by the OEM library to determine if the extension can be enabled and what capabilities it should invoke.

Major version compatibility

If the major versions of the extension-interface are different between Camera2/X and the vendor library, then it's considered incompatible and the extension is disabled.

Backward compatibility

As long as the major version is identical, Camera2/X ensures backward compatibility with OEM vendor libraries built with prior extensions-interface versions. For example, if Camera2/X supports extensions-interface 1.3.0, the OEM vendor libraries that implemented 1.0.0, 1.1.0, and 1.2.0 are still compatible. This also means that after you implement a specific version of the vendor library, Camera2/X makes sure the library is backward compatible with upcoming extension-interface versions.

Forward compatibility

Forward compatibility with vendor libraries of newer extensions-interface depends on you, the OEM. If you need some features to implement the extensions, you might want to enable the extensions starting from a certain version. In this case, you can return the supported extensions-interface version when the Camera2/X library version meets the requirements. If the Camera2/X versions aren't supported, you can return an incompatible version such as 99.0.0 to disable the extensions.

Vendor library initialization

After verifying the extensions-interface version implemented by the OEM library, Camera2/X starts the initialization process. The InitializerImpl.init() method signals to the OEM library that an app is trying to use extensions.

Camera2/X makes no other calls to the OEM library (aside from version checking) until the OEM vendor library calls OnExtensionsInitializedCallback.onSuccess() to notify the completion of initialization.

You must implement InitializerImpl as of extensions-interface 1.1.0. Camera2/X skips the library initialization step if the OEM vendor library implements extensions-interface 1.0.0.

Basic Extender versus Advanced Extender

There are two types of extensions-interface implementation: Basic Extender and Advanced Extender. Advanced Extender has been supported since extensions-interface 1.2.0.

Implement Basic Extender for extensions that process images in the camera HAL or use a post processor capable of processing YUV streams.

Implement Advanced Extender for extensions that need to customize the Camera2 stream configuration and send capture requests as needed.

See the following table for the comparison:

Basic Extender Advanced Extender
Stream configurations Fixed
Preview: PRIVATE or YUV_420_888 (if processor exists)
Still capture: JPEG or YUV_420_888 (if processor exists)		 Customizable by OEM.
Sending capture request Only Camera2/X can send capture requests. You can set the parameters to these requests. When the processor is provided for image capture, Camera2/X can send multiple capture requests and send all the images and capture results to the processor. A RequestProcessorImpl instance is provided to you to execute the camera2 capture request and get results and Image.

Camera2/X invokes startRepeating and startCapture on SessionProcessorImpl to signal the OEM to start the repeating request for preview and start the still capture sequence respectively.

Hooks in the camera pipeline
  • onPresetSession provides session parameters.
  • onEnableSession sends a single request right after CameraCaptureSession is configured.
  • onDisableSession sends a single request before CameraCaptureSession is closed.
  • initSession initializes and returns a customized camera2 session configuration for creating the capture session.
  • onCaptureSessionStart is invoked right after CameraCaptureSession is configured.
  • onCaptureSessionEnd is invoked before CameraCaptureSession is closed.
Suitable for Extensions implemented in the camera HAL or in a processor that processes YUV images.
  • Has Camera2-based implementations for the extensions.
  • Needs customized stream configuration such as RAW stream.
  • Needs Interactive capture sequence.
Supported API version Camera2 Extensions: Android 13 or higher
CameraX Extensions: camera-extensions 1.1.0 or higher		 Camera2 Extensions: Android 12L or higher
CameraX Extensions: camera-extensions 1.2.0-alpha03 or higher

App flows

The following table shows three types of app flows and their corresponding Camera Extensions API calls. While Camera2/X provide these APIs, you must properly implement the vendor library to support these flows, which we describe in more detail in a later section.

Camera2 extensions CameraX extensions
Query extension availability CameraExtensionCharacteristics . getSupportedExtensions ExtensionsManager. isExtensionAvailable
Query information CameraExtensionCharacteristics. getExtensionSupportedSizes CameraExtensionCharacteristics. getEstimatedCaptureLatencyRangeMillis CameraExtensionCharacteristics. getAvailableCaptureRequestKeys CameraExtensionCharacteristics. getAvailableCaptureResultKeys ExtensionsManager. getEstimatedCaptureLatencyRange

CameraX handles the rest of the information within the library.

Preview and still-capture with extension enabled CameraDevice. createExtensionSession

cameraExtensionsSession. setRepeatingRequest

cameraExtensionsSession. capture

 val cameraSelector = ExtensionsManager. getExtensionEnabledCameraSelector

bindToLifecycle(lifecycleOwner, cameraSelector, preview, ...)

Basic Extender

The Basic Extender interface provides hooks into several places in the camera pipeline. Each extension type has corresponding Extender classes that OEMs need to implement.

The following table lists the Extender classes OEMS need to implement for each extension:

Extender classes to implement
Night NightPreviewExtenderImpl.java

NightImageCaptureExtenderImpl.java

HDR HdrPreviewExtenderImpl.java

HdrImageCaptureExtenderImpl.java

Auto AutoPreviewExtenderImpl.java

AutoImageCaptureExtenderImpl.java

Bokeh BokehPreviewExtenderImpl.java

BokehImageCaptureExtenderImpl.java

Face retouch BeautyPreviewExtenderImpl.java

BeautyImageCaptureExtenderImpl.java

We use PreviewExtenderImpl and ImageCaptureExtenderImpl as placeholders in the following example. Replace these with the names of the actual files you're implementing.

Basic Extender has the following capabilities:

  • Inject session parameters when configuring CameraCaptureSession ( onPresetSession ).
  • Notify you of the capture session start and closing events and send a single request to notify the HAL with the returned parameters ( onEnableSession , onDisableSession ).
  • Inject capture parameters for the request ( PreviewExtenderImpl.getCaptureStage , ImageCaptureExtenderImpl.getCaptureStages ).
  • Add processors for preview and still capture that's capable of processing YUV_420_888 stream.

Let's see how Camera2/X invokes the extensions-interface to achieve the three app flows mentioned above.

App flow 1: Check extension availability

BasicExtenderAppFlow1

Figure 3. App flow 1 on Basic Extender

In this flow, Camera2/X directly calls the isExtensionAvailable() method of both PreviewExtenderImpl and ImageCaptureExtenderImpl without calling init() . Both Extender classes must return true to enable the extensions.

This is often the first step for apps to check if the given extension type is supported for a given camera ID before enabling the extension. This is because some extensions are supported only on certain camera IDs.

App flow 2: Query information

BasicExtenderAppFlow2

Figure 4. App flow 2 on Basic Extender

After determining if the extension is available, apps should query the following information before enabling the extension.

  • Still capture latency range: ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange returns the range of the capture latency for the app to evaluate if it's appropriate to enable the extension for the current scenario.

  • Supported sizes for the preview and capture surface: ImageCaptureExtenderImpl.getSupportedResolutions and PreviewExtenderImpl.getSupportedResolutions return a list of image formats and the sizes that are supported for surface format and size.

  • Supported request and result keys: Camera2/X invokes the following methods to retrieve the supported capture request keys and result keys from your implementation:

    • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys
    • ImageCaptureExtenderImpl.getAvailableCapturetResultKeys

Camera2/X always calls init() first on these Extender classes before querying for more information.

App flow 3: Preview/still capture with extension enabled (HAL implementation)

BasicExtenderAppFlow3

Figure 5. App flow 3 on Basic Extender

The above diagram illustrates the main flow of enabling preview and still capture with an extension without any processor. This means the camera HAL processes the extension.

In this flow, Camera2/X first calls init() then onInit , which notifies you that a camera session is about to start with the specified extensions. You can do heavy-lifting initialization in onInit() .

When configuring CameraCaptureSession , Camera2/X invokes onPresetSession to get the session parameters. After the capture session is configured successfully, Camera2/X invokes onEnableSession returning a CaptureStageImpl instance that contains the capture parameters. Camera2/X immediately sends a single request with these capture parameters to notify the HAL. Similarly, before the capture session is closed, Camera2/X invokes onDisableSession and then sends a single request with the returned capture parameters.

The repeating request triggered by Camera2/X contains the request parameters returned by PreviewExtenderImpl.getCaptureStage() . Furthermore, the still capture request contains the parameters returned by ImageCaptureExtenderImpl.getCaptureStages() .

Finally, Camera2/X invokes onDeInit() after the camera session has finished. You can release resources in onDeinit() .

Preview processor

In addition to the camera HAL, you can also implement extensions in a processor.

Implement PreviewExtenderImpl.getProcessorType to specify the processor type as explained below:

  • PROCESSOR_TYPE_NONE : No processor. Images are processed in the camera HAL.

  • PROCESSOR_TYPE_REQUEST_UPDATE_ONLY : The processor type lets you update the repeating request with new capture request parameters based on the latest TotalCaptureResult .

    PreviewExtenderImpl.getProcessor must return a RequestUpdateProcessorImpl instance that processes the TotalCaptureResult instance and returns a CaptureStageImpl instance to update the repeating request. PreviewExtenderImpl.getCaptureStage() should also reflect the result of the processing and return the latest CaptureStageImpl .

  • PROCESSOR_TYPE_IMAGE_PROCESSOR : This type allows you to implement a processor to process YUV_420_888 images and write the output to a PRIVATE surface.

    You need to implement and return a PreviewImageProcessorImpl instance in PreviewExtenderImpl.getProcessor . The processor is responsible for processing YUV_420_888 input images. It should write the output to the PRIVATE format of preview. Camera2/X uses a YUV_420_888 surface instead of PRIVATE to configure the CameraCaptureSession for preview.

    See following illustration for the flow:

PreviewProcessor

Figure 6. Preview flow with PreviewImageProcessorImpl

The PreviewImageProcessorImpl interface extends ProcessImpl and has three important methods:

  • onOutputSurface(Surface surface, int imageFormat) sets the output surface for the processor. For PreviewImageProcessorImpl , imageFormat is a pixel format such as PixelFormat.RGBA_8888 .

  • onResolutionUpdate(Size size) sets the size of the input image.

  • onImageFormatUpdate(int imageFormat) sets the image format of the input image. Currently, it can only be YUV_420_888 .

Image capture processor

For still capture, you can implement a processor by returning a CaptureProcessorImpl instance using ImageCaptureExtenderImpl.getCaptureProcessor . The processor is responsible to process a list of captured YUV_420_888 images and TotalCaptureResult instances and write the output to a YUV_420_888 surface.

You can safely assume that preview is enabled and running before sending the still capture request.

See the flow in the diagram below:

CaptureProcessor

Figure 7. Still capture flow with CaptureProcessorImpl

  1. Camera2/X uses a YUV_420_888 format surface for still capture to configure the capture session. Camera2/X prepares CaptureProcessorImpl by calling:

    • CaptureProcessorImpl.onImageFormatUpdate() with YUV_420_888 .
    • CaptureProcessorImpl.onResolutionUpdate() with the input image size.
    • CaptureProcessorImpl.onOutputSurface() with an output YUV_420_888 surface.

  2. ImageCaptureExtenderImpl.getCaptureStages returns a list of CaptureStageImpl , where each element maps to a CaptureRequest instance with capture parameters that are sent by Camera2/X. For example, if it returns a list of three CaptureStageImpl instances, Camera2/X sends three capture requests with corresponding capture parameters using the captureBurst API.

  3. The received images and TotalCaptureResult instances are bundled together and sent to CaptureProcessorImpl for processing.

  4. CaptureProcessorImpl writes the result Image ( YUV_420_888 format) to the output surface specified by the onOutputSurface() call. Camera2/X converts it into JPEG images if necessary.

Support capture request keys and results

In addition to camera preview and capture, apps can set zoom, flash parameters, or trigger a tap-to-focus. These parameters might not be compatible with your extension implementation.

The following methods have been added to extensions-interface 1.3.0 to allow you to expose the parameters that your implementation supports:

  • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys() returns the capture request keys supported by your implementation.
  • ImageCaptureExtenderImpl.getAvailableCaptureResultKeys() returns the capture result keys that are contained in the capture result.

If the camera HAL processes the extension, Camera2/X retrieves the capture results in CameraCaptureSession.CaptureCallback . However, if the processor is implemented, then Camera2/X retrieves the capture results in ProcessResultImpl , which is passed to the process() method in PreviewImageProcessorImpl and CaptureProcessorImpl . You're responsible for reporting the capture result through ProcessResultImpl to Camera2/X.

See the definition of the CaptureProcessorImpl interface below as an example. In extensions-interface 1.3.0 or higher, the second process() call is invoked:

Interface CaptureProcessorImpl extends ProcessorImpl {
    // invoked when extensions-interface version < 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results);
    // invoked when extensions-interface version >= 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results,
            ProcessResultImpl resultCallback, Executor executor);
}

For common camera operations like zoom, tap-to-focus, flash, and exposure compensation, we recommend supporting the following keys for both capture request and capture result:

  • Zoom:
    • CaptureRequest#CONTROL_ZOOM_RATIO
    • CaptureRequest#SCALER_CROP_REGION
  • Tap-to-focus:
    • CaptureRequest#CONTROL_AF_MODE
    • CaptureRequest#CONTROL_AF_TRIGGER
    • CaptureRequest#CONTROL_AF_REGIONS
    • CaptureRequest#CONTROL_AE_REGIONS
    • CaptureRequest#CONTROL_AWB_REGIONS
  • Flash:
    • CaptureRequest#CONTROL_AE_MODE
    • CaptureRequest#CONTROL_AE_PRECAPTURE_TRIGGER
    • CaptureRequest#FLASH_MODE
  • Exposure compensation:
    • CaptureRequest#CONTROL_AE_EXPOSURE_COMPENSATION

For Basic Extenders that implement 1.2.0 or prior versions, the CameraX Extensions API explicitly supports all the above keys. For extensions-interface 1.3.0, both CameraX and Camera2 honor the returned list and support only the keys contained in it. For example, if you decide to return only CaptureRequest#CONTROL_ZOOM_RATIO and CaptureRequest#SCALER_CROP_REGION in the 1.3.0 implementation, then that means only zoom is supported for the app while tap-to-focus, flash, and exposure compensation aren't allowed.

Advanced Extender

Advanced Extender is a type of vendor implementation based on the Camera2 API. This Extender type was added in extensions-interface 1.2.0. Depending on the device manufacturer, extensions might be implemented in the app layer, which depends on the following factors:

  • Custom stream configuration: Configure custom streams like RAW stream or have multiple streams for different physical camera IDs.

  • Capability to send Camera2 requests: Support a complicated interaction logic that can send capture requests with parameters based on the results of previous requests.

Advanced Extender provides a wrapper, or an intermediate layer, so you can customize the stream configuration and send capture requests on demand.

Files to implement

To switch to the Advanced Extender implementation, the isAdvancedExtenderImplemented() method in ExtensionVersionImpl must return true . For each extension type, OEMs must implement the corresponding Extender classes. The Advanced Extender implementation files are in the advanced package.

Extender classes to implement
Night advanced/NightAdvancedExtenderImpl.java
HDR advanced/HdrAdvancedExtenderImpl.java
Auto advanced/AutoAdvancedExtenderImpl.java
Bokeh advanced/BokehAdvancedExtenderImpl.java
Face Retouch advanced/BeautyAdvancedExtenderImpl.java

We use AdvancedExtenderImpl as a placeholder in the following example. Replace it with the name of the Extender file for the extension you're implementing.

Let's see how Camera2/X invokes the extensions-interface to achieve the three app flows.

App flow 1: Check extensions availability

AdvancedAppFlow1

Figure 8. App flow 1 on Advanced Extender

First, the app checks if the given extension is supported.

App flow 2: Query information

AdvancedAppFlow2

Figure 9. App flow 2 on Advanced Extender

After calling AdvancedExtenderImpl.init() , the app can query the following the information on AdvancedExtenderImpl :

  • Estimated still capture latency: AdvancedExtenderImpl.getEstimatedCaptureLatencyRange() returns the range of the capture latency for the app to evaluate if it is appropriate to enable the extension for the current scenario.

  • Supported resolutions for preview and still capture:

    • AdvancedExtenderImpl.getSupportedPreviewOutputResolutions() returns a map of image format to the sizes list that are supported for preview surface format and size. OEMs must support at least the PRIVATE format.

    • AdvancedExtenderImpl.getSupportedCaptureOutputResolutions() returns the supported format and sizes for still capture surface. OEMs must support both JPEG and YUV_420_888 format output.

    • AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() returns the supported sizes for an extra YUV_420_888 stream for image analysis. If the image analysis YUV surface isn't supported, getSupportedYuvAnalysisResolutions() should return null or an empty list.

  • Available capture request keys/results (added in extensions-interface 1.3.0): Camera2/X invokes the following methods to retrieve the supported capture request keys and result keys from your implementation:

    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys
    • AdvancedExtenderImpl.getAvailableCaptureResultKeys

For more information, see Support capture request keys and results .

App flow 3: Preview/still capture with extension enabled

AdvancedAppFlow3

Figure 10. App flow 3 on Advanced Extender

The above diagram shows the main flow for starting preview and still capture for the Advanced Extender type. Let's walk through each step.

  1. SessionProcessorImpl instance

    The core Advanced Extender implementation is in SessionProcessorImpl , which is responsible for providing customized session configuration and sending capture requests to initiate the preview and still capture request. AdvancedExtenderImpl.createSessionProcessor() is invoked to return the SessionProcessorImpl instance.

  2. initSession

    SessionProcessorImpl.initSession() initializes the session for the extension. This is where you allocate resources and return a session configuration for preparing a CameraCaptureSession .

    For the input parameters, Camera2/X specifies the output surface configurations for preview, still capture, and an optional YUV image analysis. This output surface configuration ( OutputSurfaceImpl ) contains the surface, size and image format that are retrieved by following methods in AdvancedExtenderImpl :

    • getSupportedPreviewOutputResolutions()
    • getSupportedCaptureOutputResolutions()
    • getSupportedYuvAnalysisResolutions()

    You must return a Camera2SessionConfigImpl instance, which consists of a list of Camera2OutputConfigImpl instances and the session parameters used for configuring CameraCaptureSession . You're responsible for outputting the correct camera images to the output surfaces passed in by Camera2/X. Here are some options to enable the output:

    • Processing in camera HAL: You can directly add the output surfaces to CameraCaptureSession with a SurfaceOutputConfigImpl implementation. This configures the supplied output surface to the camera pipeline and allows the camera HAL to process the image.

    • Processing intermediate ImageReader surface (RAW, YUV, etc): Add the intermediate ImageReader surfaces to the CameraCaptureSession with an ImageReaderOutputConfigImpl instance.

      You need to process the intermediate images and write the result image to the output surface.

    • Use Camera2 surface sharing: Use surface sharing with another surface by adding any Camera2OutputConfigImpl instance to the getSurfaceSharingOutputConfigs() method of another Camera2OutputConfigImpl instance. The surface format and size must be identical.

    All Camera2OutputConfigImpl including SurfaceOutputConfigImpl and ImageReaderOutputConfigImpl must have a unique ID ( getId() ), which is used to specify the target surface and retrieve the image from ImageReaderOutputConfigImpl .

  3. onCaptureSessionStart and RequestProcessorImpl

    When CameraCaptureSession starts and the Camera framework invokes onConfigured() , then Camera2/X invokes SessionProcessorImpl.onCaptureSessionStart() with the Camera2 request wrapper RequestProcessImpl . Camera2/X implements RequestProcessImpl , which enables you to execute the capture requests , and retrieve images if ImageReaderOutputConfigImpl is used.

    The RequestProcessImpl APIs are similar to the Camera2 CameraCaptureSession APIs in terms of executing requests. The differences are:

    • The target surface is specified by the ID of the Camera2OutputConfigImpl instance.
    • The capability of retrieving the image of the ImageReader .

    You can call RequestProcessorImpl.setImageProcessor() with a specified Camera2OutputConfigImpl ID to register an ImageProcessorImpl instance to receive images.

    The RequestProcessImpl instance becomes invalid after Camera2/X calls SessionProcessorImpl.onCaptureSessionEnd() .

  4. Start the preview and take a picture

    In the Advanced Extender implementation, you can send capture requests through the RequestProcessorImpl interface. Camera2/X notifies you to start the repeating request for preview or the still capture sequence by calling SessionProcessorImpl#startRepeating and SessionProcessorImpl#startCapture respectively. You should send capture requests to satisfy these preview and still-capture requests.

    Camera2/X also sets the capture request parameters through SessionProcessorImpl#setParameters . You must set these request parameters (if parameters are supported) on both the repeating and single requests.

    You must support at least CaptureRequest.JPEG_ORIENTATION and CaptureRequest.JPEG_QUALITY . extensions-interface 1.3.0 supports request and result keys, which are exposed by the following methods:

    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys()
    • AdvancedExtenderImpl.getAvailableCaptureResultKeys()

    When developers set the keys in the getAvailableCaptureRequestKeys list, you must enable the parameters and ensure the capture result contains the keys in the getAvailableCaptureResultKeys list.

  5. startTrigger

    SessionProcessorImpl.startTrigger() is invoked to start the trigger such as CaptureRequest.CONTROL_AF_TRIGGER and CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER . You can disregard any capture request keys that weren't advertised in AdvancedExtenderImpl.getAvailableCaptureRequestKeys() .

    startTrigger() has been supported since extensions-interface 1.3.0. It enables apps to implement tap-to-focus and flash with extensions.

  6. Clean up

    When finishing a capture session, SessionProcessorImpl.onCaptureSessionEnd() is invoked ahead of closing CameraCaptureSession . After the capture session has closed, deInitSession() performs the clean up.

Support preview, still capture, and image analysis

You should apply the extension for both the preview and still capture use cases. However, if the latency is too high to smoothly show the preview, you can apply the extension only for still capture.

For the Basic Extender type, regardless of enabling the extension for preview, you must implement both ImageCaptureExtenderImpl and PreviewExtenderImpl for a given extension. Often, an app also uses a YUV stream to analyze the image content such as finding QR codes or text. To better support this use case , you should support the stream combination of preview, still capture, and a YUV_420_888 stream for configuring CameraCaptureSession . This means that if you implement a processor, then you have to support the stream combination of three YUV_420_888 streams.

For Advanced Extender, Camera2/X passes three output surfaces to the SessionProcessorImpl.initSession() call. These output surfaces are for preview , still capture, and image analysis, respectively. You must ensure that preview and still capture output surfaces show the valid output. However, for the image analysis output surface, ensure it's working only when it's non-null. If your implementation can't support the image analysis stream, you can return an empty list in AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() . This ensures the image analysis output surface is always null in SessionProcessorImpl.initSession() .

Support video capture

The current Camera Extension architecture supports only the preview and still capture use cases. We don't support enabling the extension on the MediaCodec or MediaRecorder surfaces for recording the video. However, it's possible for apps to record the preview output.

Supporting MediaCodec and MediaRecorder surfaces is under investigation.

Extensions interface version history

The following table shows the Camera Extension interface version history. You should always implement the vendor library with the latest version.

Version Added features
1.0.0
  • Version verification
    • ExtensionVersionImpl
  • Basic Extender
    • PreviewExtenderImpl
    • ImageCaptureExtenderImpl
    • Processor
      • PreviewImageProcessorImpl
      • CaptureProcessorImpl
      • RequestUpdateProcessorImpl
1.1.0
  • Library initialization
    • InitializerImpl
  • Expose supported resolutions
    • PreviewExtenderImpl.getSupportedResolutions
    • ImageCaptureExtenderImpl.getSupportedResolutions
1.2.0
  • AdvancedExtender
    • AdvancedExtenderImpl
    • SessionProcessorImpl
  • Get estimated capture latency
    • ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange
1.3.0
  • Expose supported capture request keys/results keys
    • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys
    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys
    • New process() call that takes ProcessResultImpl in PreviewImageProcessorImpl and CaptureProcessorImpl
    • Support trigger type request
      • AdvancedExtenderImpl.startTrigger

Reference implementation

The following reference OEM vendor library implementations are available in frameworks/ex .

  • advancedSample : A basic implementation of Advanced Extender.

  • sample : A basic implementation of Basic Extender.

  • service_based_sample : An implementation that demonstrates how to host Camera Extensions in a Service . This implementation contains the following components:

    • oem_library : A Camera Extensions OEM library for Camera2 and CameraX Extensions APIs that implements Extensions-Interface . This acts as a passthrough that forwards calls from Extensions-Interface to the service. This library also provides AIDL files and wrapper classes to communicate with the service.

      Advanced Extender is enabled by default. To enable the Basic Extender, change ExtensionsVersionImpl#isAdvancedExtenderImplemented to return false .

    • extensions_service : A sample implementation of the Extensions Service. Add your implementation here. The interface to implement in the service is similar to the Extensions-Interface . For example, implementing the IAdvancedExtenderImpl.Stub performs the same operations as AdvancedExtenderImpl . ImageWrapper and TotalCaptureResultWrapper are required to make Image and TotalCaptureResult parcelable.

Set up the vendor library on a device

The OEM vendor library isn't built into an app; it's loaded from the device at runtime by Camera2/X. In CameraX, the <uses-library> tag declares that the androidx.camera.extensions.impl library, which is defined in the AndroidManifest.xml file of the camera-extensions library, is a dependency of CameraX and must be loaded at runtime. In Camera2, the framework loads an extensions service that also declares that the <uses-library> loads the same androidx.camera.extensions.impl library at runtime.

This allows third-party apps using extensions to automatically load the OEM vendor library. The OEM library is marked as optional so apps can run on devices that don't have the library on the device. Camera2/X handles this behavior automatically when an app tries to use a camera extension as long as the device manufacturer places the OEM library on the device so that it can be discovered by the app.

To set up the OEM library on a device, do the following:

  1. Add a permission file, which is required by the <uses-library> tag, using the following format: /etc/permissions/ ANY_FILENAME .xml . For example, /etc/permissions/camera_extensions.xml . The files in this directory provide a mapping of the library named in <uses-library> to the actual file path on the device.

  2. Use the example below to add the required information to the file.

    • name must be androidx.camera.extensions.impl as that's the library that CameraX searches for.
    • file is the absolute path of the file that contains the extensions implementation (for example, /system/framework/androidx.camera.extensions.impl.jar ).
    <?xml version="1.0" encoding="utf-8"?>
<permissions>
    <library name="androidx.camera.extensions.impl"
             file="OEM_IMPLEMENTED_JAR" />
</permissions>

In Android 12 or higher, devices supporting CameraX extensions must have the ro.camerax.extensions.enabled property set to true , which allows for querying whether a device supports extensions. To do this, add the following line in the device make file:

PRODUCT_VENDOR_PROPERTIES += \
    ro.camerax.extensions.enabled=true \

Validation

To test your implementation of the OEM vendor library during the development stage, use the example app at androidx-main/camera/integration-tests/extensionstestapp/ , which runs through various vendor extensions.

After you complete your implementation, use the Camera Extensions Validation Tool to run automated and manual tests to verify that the vendor library is implemented correctly.

Extended scene mode versus Camera Extensions

For the bokeh extension, in addition to exposing it using Camera Extensions, you can expose the extension using the extended scene mode, which is enabled through the CONTROL_EXTENDED_SCENE_MODE key. For more implementation details, see Camera Bokeh .

Extended scene mode has fewer restrictions compared to Camera Extensions for camera2 apps. For example, you can enable extended scene mode in a regular CameraCaptureSession instance that supports flexible stream combinations and capture request parameters. In contrast, camera extensions support only a fixed set of stream types and have limited support for capture request parameters.

A downside of extended scene mode is that you can only implement it in the camera HAL, which means that it must be verified to work across all orthogonal controls available to app developers.

We recommend exposing bokeh using both the extended scene mode and Camera Extensions because apps might prefer to use a particular API to enable bokeh. We recommend first using the extended scene mode because this is the most flexible way for apps to enable the bokeh extension. Then you can implement the camera extensions interface based on the extended scene mode. If implementing bokeh in the camera HAL is difficult, for example, because it requires a post processor running in the app layer to process images, we recommend implementing the bokeh extension using the Camera Extensions interface.

Frequently asked questions (FAQs)

Are there any restrictions on API levels?

Yes. This depends on the Android API feature set that's required by the OEM vendor library implementation. For example, ExtenderStateListener.onPresetSession() uses the SessionConfiguration.setSessionParameters() call to set a baseline set of tags. This call is available only on API level 28 and higher. For details on specific interface methods, see the API reference documentation .

,

Device manufacturers can expose extensions such as bokeh, night mode, and HDR to third-party developers through the Camera Extensions interface provided by the OEM vendor library. Developers can use the Camera2 Extensions API and the CameraX Extensions API to access the extensions implemented in the OEM vendor library.

For a list of supported extensions, which is the same across Camera2 and CameraX, see CameraX Extensions API . If you want to add an extension, file a bug with the Issue Tracker .

This page describes how to implement and enable the OEM vendor library on devices.

Architecture

The following diagram describes the architecture of the Camera Extensions interface or extensions-interface : Architecture

Figure 1. Camera Extensions architecture diagram

As shown in the diagram, to support Camera Extensions, you need to implement the extensions-interface provided by the OEM vendor library. Your OEM vendor library enables two APIs: CameraX Extensions API and Camera2 Extensions API , which are used by CameraX and Camera2 apps, respectively, to access vendor extensions.

Implement the OEM vendor library

To implement the OEM vendor library, copy the camera-extensions-stub files into a system library project. These files define the Camera Extensions interface.

The camera-extensions-stub files are divided into the following categories:

Essential interface files (don't modify)

  • PreviewExtenderImpl.java
  • ImageCaptureExtenderImpl.java
  • ExtenderStateListener.java
  • ProcessorImpl.java
  • PreviewImageProcessorImpl.java
  • CaptureProcessorImpl.java
  • CaptureStageImpl.java
  • RequestUpdateProcessorImpl.java
  • ProcessResultImpl.java
  • advanced/AdvancedExtenderImpl.java
  • advanced/Camera2OutputConfigImpl.java
  • advanced/Camera2SessionConfigImpl.java
  • advanced/ImageProcessorImpl.java
  • advanced/ImageReaderOutputConfigImpl.java
  • advanced/ImageReferenceImpl.java
  • advanced/MultiResolutionImageReaderOutputConfigImpl.java
  • advanced/OutputSurfaceImpl.java
  • advanced/RequestProcessorImpl.java
  • advanced/SessionProcessorImpl.java
  • advanced/SurfaceOutputConfigImpl.java

Mandatory implementations (add your implementation)

  • ExtensionVersionImpl.java
  • InitializerImpl.java

Bokeh extender classes (implement it if Bokeh extension is supported)

  • BokehImageCaptureExtenderImpl.java
  • BokehPreviewExtenderImpl.java
  • advanced/BokehAdvancedExtenderImpl.java

Night extender classes (implement it if Night extension is supported)

  • NightImageCaptureExtenderImpl.java
  • NightPreviewExtenderImpl.java
  • advanced/NightAdvancedExtenderImpl.java

Auto extender classes (implement it if Auto extension is supported)

  • AutoImageCaptureExtenderImpl.java
  • AutoPreviewExtenderImpl.java
  • advanced/AutoAdvancedExtenderImpl.java

HDR extender classes (implement it if HDR extension is supported)

  • HdrImageCaptureExtenderImpl.java
  • HdrPreviewExtenderImpl.java
  • advanced/HdrAdvancedExtenderImpl.java

Face Retouch extender classes (implement it if Face Retouch extension is supported)

  • BeautyImageCaptureExtenderImpl.java
  • BeautyPreviewExtenderImpl.java
  • advanced/BeautyAdvancedExtenderImpl.java

Utilities (optional, can be deleted)

  • advanced/Camera2OutputConfigImplBuilder.java
  • advanced/Camera2SessionConfigImplBuilder.java

You aren't required to provide an implementation for every extension. If you don't implement an extension, set isExtensionAvailable() to return false or remove the corresponding Extender classes. The Camera2 and CameraX Extensions APIs report to the app that the extension is unavailable.

Let's walk through how the Camera2 and CameraX Extensions APIs interact with the vendor library to enable an extension. The following diagram illustrates the end-to-end flow using the Night extension as an example:

Mainflow

Figure 2. Night extension implementation

  1. Version verification:

    Camera2/X calls ExtensionVersionImpl.checkApiVersion() to ensure that the OEM-implemented extensions-interface version is compatible with Camera2/X supported versions.

  2. Vendor library initialization:

    InitializerImpl has a method init() that initializes the vendor library. Camera2/X completes the initialization before accessing the Extender classes.

  3. Instantiate Extender classes:

    Instantiates the Extender classes for the extension. There are two Extender types: Basic Extender and Advanced Extender. You must implement one Extender type for all Extensions. For more information, see Basic Extender versus Advanced Extender .

    Camera2/X instantiates and interacts with the Extender classes to retrieve information and enable the extension. For a given extension, Camera2/X can instantiate the Extender classes multiple times. As a result, don't do heavy-lifting initialization in the constructor or the init() call. Do the heavy lifting only when the camera session is about to start, such as when onInit() is called in Basic Extender or initSession() is called in Advanced Extender.

    For the Night extension, the following Extender classes are instantiated for the Basic Extender type:

    • NightImageCaptureExtenderImpl.java
    • NightPreviewExtenderImpl.java

    And for the Advanced Extender type:

    • NightAdvancedExtenderImpl.java

  4. Check extension availability:

    Before enabling the extension, isExtensionAvailable() checks if the extension is available on the specified camera ID through the Extender instance.

  5. Initialize the Extender with camera information:

    Camera2/X calls init() on the Extender instance and passes it the camera ID and CameraCharacteristics .

  6. Query information:

    Invokes the Extender class to retrieve information such as supported resolutions, still capture estimated latency, and capture request keys from the Extender in preparation for enabling the extension.

  7. Enable extension on the Extender:

    The Extender class provides all the interfaces needed to enable the class. It offers a mechanism to hook OEM implementation into the Camera2 pipeline such as injecting capture request parameters or enabling a post processor.

    For the Advanced Extender type, Camera2/X interacts with SessionProcessorImpl to enable the extension. Camera2/X retrieves the SessionProcessorImpl instance by calling createSessionProcessor() on the Extender.

The following sections describe the extension flow in greater detail.

Version verification

When loading the OEM vendor library from the device at runtime, Camera2/X verifies if the library is compatible with the extensions-interface version. The extensions-interface uses semantic versioning, or MAJOR.MINOR.PATCH, for example, 1.1.0 or 1.2.0. However, only the major and minor versions are used during the version verification.

To verify the version, Camera2/X calls ExtensionVersionImpl.checkApiVersion() with the supported extensions-interface version. Camera2/X then uses the version reported by the OEM library to determine if the extension can be enabled and what capabilities it should invoke.

Major version compatibility

If the major versions of the extension-interface are different between Camera2/X and the vendor library, then it's considered incompatible and the extension is disabled.

Backward compatibility

As long as the major version is identical, Camera2/X ensures backward compatibility with OEM vendor libraries built with prior extensions-interface versions. For example, if Camera2/X supports extensions-interface 1.3.0, the OEM vendor libraries that implemented 1.0.0, 1.1.0, and 1.2.0 are still compatible. This also means that after you implement a specific version of the vendor library, Camera2/X makes sure the library is backward compatible with upcoming extension-interface versions.

Forward compatibility

Forward compatibility with vendor libraries of newer extensions-interface depends on you, the OEM. If you need some features to implement the extensions, you might want to enable the extensions starting from a certain version. In this case, you can return the supported extensions-interface version when the Camera2/X library version meets the requirements. If the Camera2/X versions aren't supported, you can return an incompatible version such as 99.0.0 to disable the extensions.

Vendor library initialization

After verifying the extensions-interface version implemented by the OEM library, Camera2/X starts the initialization process. The InitializerImpl.init() method signals to the OEM library that an app is trying to use extensions.

Camera2/X makes no other calls to the OEM library (aside from version checking) until the OEM vendor library calls OnExtensionsInitializedCallback.onSuccess() to notify the completion of initialization.

You must implement InitializerImpl as of extensions-interface 1.1.0. Camera2/X skips the library initialization step if the OEM vendor library implements extensions-interface 1.0.0.

Basic Extender versus Advanced Extender

There are two types of extensions-interface implementation: Basic Extender and Advanced Extender. Advanced Extender has been supported since extensions-interface 1.2.0.

Implement Basic Extender for extensions that process images in the camera HAL or use a post processor capable of processing YUV streams.

Implement Advanced Extender for extensions that need to customize the Camera2 stream configuration and send capture requests as needed.

See the following table for the comparison:

Basic Extender Advanced Extender
Stream configurations Fixed
Preview: PRIVATE or YUV_420_888 (if processor exists)
Still capture: JPEG or YUV_420_888 (if processor exists)		 Customizable by OEM.
Sending capture request Only Camera2/X can send capture requests. You can set the parameters to these requests. When the processor is provided for image capture, Camera2/X can send multiple capture requests and send all the images and capture results to the processor. A RequestProcessorImpl instance is provided to you to execute the camera2 capture request and get results and Image.

Camera2/X invokes startRepeating and startCapture on SessionProcessorImpl to signal the OEM to start the repeating request for preview and start the still capture sequence respectively.

Hooks in the camera pipeline
  • onPresetSession provides session parameters.
  • onEnableSession sends a single request right after CameraCaptureSession is configured.
  • onDisableSession sends a single request before CameraCaptureSession is closed.
  • initSession initializes and returns a customized camera2 session configuration for creating the capture session.
  • onCaptureSessionStart is invoked right after CameraCaptureSession is configured.
  • onCaptureSessionEnd is invoked before CameraCaptureSession is closed.
Suitable for Extensions implemented in the camera HAL or in a processor that processes YUV images.
  • Has Camera2-based implementations for the extensions.
  • Needs customized stream configuration such as RAW stream.
  • Needs Interactive capture sequence.
Supported API version Camera2 Extensions: Android 13 or higher
CameraX Extensions: camera-extensions 1.1.0 or higher		 Camera2 Extensions: Android 12L or higher
CameraX Extensions: camera-extensions 1.2.0-alpha03 or higher

App flows

The following table shows three types of app flows and their corresponding Camera Extensions API calls. While Camera2/X provide these APIs, you must properly implement the vendor library to support these flows, which we describe in more detail in a later section.

Camera2 extensions CameraX extensions
Query extension availability CameraExtensionCharacteristics . getSupportedExtensions ExtensionsManager. isExtensionAvailable
Query information CameraExtensionCharacteristics. getExtensionSupportedSizes CameraExtensionCharacteristics. getEstimatedCaptureLatencyRangeMillis CameraExtensionCharacteristics. getAvailableCaptureRequestKeys CameraExtensionCharacteristics. getAvailableCaptureResultKeys ExtensionsManager. getEstimatedCaptureLatencyRange

CameraX handles the rest of the information within the library.

Preview and still-capture with extension enabled CameraDevice. createExtensionSession

cameraExtensionsSession. setRepeatingRequest

cameraExtensionsSession. capture

 val cameraSelector = ExtensionsManager. getExtensionEnabledCameraSelector

bindToLifecycle(lifecycleOwner, cameraSelector, preview, ...)

Basic Extender

The Basic Extender interface provides hooks into several places in the camera pipeline. Each extension type has corresponding Extender classes that OEMs need to implement.

The following table lists the Extender classes OEMS need to implement for each extension:

Extender classes to implement
Night NightPreviewExtenderImpl.java

NightImageCaptureExtenderImpl.java

HDR HdrPreviewExtenderImpl.java

HdrImageCaptureExtenderImpl.java

Auto AutoPreviewExtenderImpl.java

AutoImageCaptureExtenderImpl.java

Bokeh BokehPreviewExtenderImpl.java

BokehImageCaptureExtenderImpl.java

Face retouch BeautyPreviewExtenderImpl.java

BeautyImageCaptureExtenderImpl.java

We use PreviewExtenderImpl and ImageCaptureExtenderImpl as placeholders in the following example. Replace these with the names of the actual files you're implementing.

Basic Extender has the following capabilities:

  • Inject session parameters when configuring CameraCaptureSession ( onPresetSession ).
  • Notify you of the capture session start and closing events and send a single request to notify the HAL with the returned parameters ( onEnableSession , onDisableSession ).
  • Inject capture parameters for the request ( PreviewExtenderImpl.getCaptureStage , ImageCaptureExtenderImpl.getCaptureStages ).
  • Add processors for preview and still capture that's capable of processing YUV_420_888 stream.

Let's see how Camera2/X invokes the extensions-interface to achieve the three app flows mentioned above.

App flow 1: Check extension availability

BasicExtenderAppFlow1

Figure 3. App flow 1 on Basic Extender

In this flow, Camera2/X directly calls the isExtensionAvailable() method of both PreviewExtenderImpl and ImageCaptureExtenderImpl without calling init() . Both Extender classes must return true to enable the extensions.

This is often the first step for apps to check if the given extension type is supported for a given camera ID before enabling the extension. This is because some extensions are supported only on certain camera IDs.

App flow 2: Query information

BasicExtenderAppFlow2

Figure 4. App flow 2 on Basic Extender

After determining if the extension is available, apps should query the following information before enabling the extension.

  • Still capture latency range: ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange returns the range of the capture latency for the app to evaluate if it's appropriate to enable the extension for the current scenario.

  • Supported sizes for the preview and capture surface: ImageCaptureExtenderImpl.getSupportedResolutions and PreviewExtenderImpl.getSupportedResolutions return a list of image formats and the sizes that are supported for surface format and size.

  • Supported request and result keys: Camera2/X invokes the following methods to retrieve the supported capture request keys and result keys from your implementation:

    • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys
    • ImageCaptureExtenderImpl.getAvailableCapturetResultKeys

Camera2/X always calls init() first on these Extender classes before querying for more information.

App flow 3: Preview/still capture with extension enabled (HAL implementation)

BasicExtenderAppFlow3

Figure 5. App flow 3 on Basic Extender

The above diagram illustrates the main flow of enabling preview and still capture with an extension without any processor. This means the camera HAL processes the extension.

In this flow, Camera2/X first calls init() then onInit , which notifies you that a camera session is about to start with the specified extensions. You can do heavy-lifting initialization in onInit() .

When configuring CameraCaptureSession , Camera2/X invokes onPresetSession to get the session parameters. After the capture session is configured successfully, Camera2/X invokes onEnableSession returning a CaptureStageImpl instance that contains the capture parameters. Camera2/X immediately sends a single request with these capture parameters to notify the HAL. Similarly, before the capture session is closed, Camera2/X invokes onDisableSession and then sends a single request with the returned capture parameters.

The repeating request triggered by Camera2/X contains the request parameters returned by PreviewExtenderImpl.getCaptureStage() . Furthermore, the still capture request contains the parameters returned by ImageCaptureExtenderImpl.getCaptureStages() .

Finally, Camera2/X invokes onDeInit() after the camera session has finished. You can release resources in onDeinit() .

Preview processor

In addition to the camera HAL, you can also implement extensions in a processor.

Implement PreviewExtenderImpl.getProcessorType to specify the processor type as explained below:

  • PROCESSOR_TYPE_NONE : No processor. Images are processed in the camera HAL.

  • PROCESSOR_TYPE_REQUEST_UPDATE_ONLY : The processor type lets you update the repeating request with new capture request parameters based on the latest TotalCaptureResult .

    PreviewExtenderImpl.getProcessor must return a RequestUpdateProcessorImpl instance that processes the TotalCaptureResult instance and returns a CaptureStageImpl instance to update the repeating request. PreviewExtenderImpl.getCaptureStage() should also reflect the result of the processing and return the latest CaptureStageImpl .

  • PROCESSOR_TYPE_IMAGE_PROCESSOR : This type allows you to implement a processor to process YUV_420_888 images and write the output to a PRIVATE surface.

    You need to implement and return a PreviewImageProcessorImpl instance in PreviewExtenderImpl.getProcessor . The processor is responsible for processing YUV_420_888 input images. It should write the output to the PRIVATE format of preview. Camera2/X uses a YUV_420_888 surface instead of PRIVATE to configure the CameraCaptureSession for preview.

    See following illustration for the flow:

PreviewProcessor

Figure 6. Preview flow with PreviewImageProcessorImpl

The PreviewImageProcessorImpl interface extends ProcessImpl and has three important methods:

  • onOutputSurface(Surface surface, int imageFormat) sets the output surface for the processor. For PreviewImageProcessorImpl , imageFormat is a pixel format such as PixelFormat.RGBA_8888 .

  • onResolutionUpdate(Size size) sets the size of the input image.

  • onImageFormatUpdate(int imageFormat) sets the image format of the input image. Currently, it can only be YUV_420_888 .

Image capture processor

For still capture, you can implement a processor by returning a CaptureProcessorImpl instance using ImageCaptureExtenderImpl.getCaptureProcessor . The processor is responsible to process a list of captured YUV_420_888 images and TotalCaptureResult instances and write the output to a YUV_420_888 surface.

You can safely assume that preview is enabled and running before sending the still capture request.

See the flow in the diagram below:

CaptureProcessor

Figure 7. Still capture flow with CaptureProcessorImpl

  1. Camera2/X uses a YUV_420_888 format surface for still capture to configure the capture session. Camera2/X prepares CaptureProcessorImpl by calling:

    • CaptureProcessorImpl.onImageFormatUpdate() with YUV_420_888 .
    • CaptureProcessorImpl.onResolutionUpdate() with the input image size.
    • CaptureProcessorImpl.onOutputSurface() with an output YUV_420_888 surface.

  2. ImageCaptureExtenderImpl.getCaptureStages returns a list of CaptureStageImpl , where each element maps to a CaptureRequest instance with capture parameters that are sent by Camera2/X. For example, if it returns a list of three CaptureStageImpl instances, Camera2/X sends three capture requests with corresponding capture parameters using the captureBurst API.

  3. The received images and TotalCaptureResult instances are bundled together and sent to CaptureProcessorImpl for processing.

  4. CaptureProcessorImpl writes the result Image ( YUV_420_888 format) to the output surface specified by the onOutputSurface() call. Camera2/X converts it into JPEG images if necessary.

Support capture request keys and results

In addition to camera preview and capture, apps can set zoom, flash parameters, or trigger a tap-to-focus. These parameters might not be compatible with your extension implementation.

The following methods have been added to extensions-interface 1.3.0 to allow you to expose the parameters that your implementation supports:

  • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys() returns the capture request keys supported by your implementation.
  • ImageCaptureExtenderImpl.getAvailableCaptureResultKeys() returns the capture result keys that are contained in the capture result.

If the camera HAL processes the extension, Camera2/X retrieves the capture results in CameraCaptureSession.CaptureCallback . However, if the processor is implemented, then Camera2/X retrieves the capture results in ProcessResultImpl , which is passed to the process() method in PreviewImageProcessorImpl and CaptureProcessorImpl . You're responsible for reporting the capture result through ProcessResultImpl to Camera2/X.

See the definition of the CaptureProcessorImpl interface below as an example. In extensions-interface 1.3.0 or higher, the second process() call is invoked:

Interface CaptureProcessorImpl extends ProcessorImpl {
    // invoked when extensions-interface version < 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results);
    // invoked when extensions-interface version >= 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results,
            ProcessResultImpl resultCallback, Executor executor);
}

For common camera operations like zoom, tap-to-focus, flash, and exposure compensation, we recommend supporting the following keys for both capture request and capture result:

  • Zoom:
    • CaptureRequest#CONTROL_ZOOM_RATIO
    • CaptureRequest#SCALER_CROP_REGION
  • Tap-to-focus:
    • CaptureRequest#CONTROL_AF_MODE
    • CaptureRequest#CONTROL_AF_TRIGGER
    • CaptureRequest#CONTROL_AF_REGIONS
    • CaptureRequest#CONTROL_AE_REGIONS
    • CaptureRequest#CONTROL_AWB_REGIONS
  • Flash:
    • CaptureRequest#CONTROL_AE_MODE
    • CaptureRequest#CONTROL_AE_PRECAPTURE_TRIGGER
    • CaptureRequest#FLASH_MODE
  • Exposure compensation:
    • CaptureRequest#CONTROL_AE_EXPOSURE_COMPENSATION

For Basic Extenders that implement 1.2.0 or prior versions, the CameraX Extensions API explicitly supports all the above keys. For extensions-interface 1.3.0, both CameraX and Camera2 honor the returned list and support only the keys contained in it. For example, if you decide to return only CaptureRequest#CONTROL_ZOOM_RATIO and CaptureRequest#SCALER_CROP_REGION in the 1.3.0 implementation, then that means only zoom is supported for the app while tap-to-focus, flash, and exposure compensation aren't allowed.

Advanced Extender

Advanced Extender is a type of vendor implementation based on the Camera2 API. This Extender type was added in extensions-interface 1.2.0. Depending on the device manufacturer, extensions might be implemented in the app layer, which depends on the following factors:

  • Custom stream configuration: Configure custom streams like RAW stream or have multiple streams for different physical camera IDs.

  • Capability to send Camera2 requests: Support a complicated interaction logic that can send capture requests with parameters based on the results of previous requests.

Advanced Extender provides a wrapper, or an intermediate layer, so you can customize the stream configuration and send capture requests on demand.

Files to implement

To switch to the Advanced Extender implementation, the isAdvancedExtenderImplemented() method in ExtensionVersionImpl must return true . For each extension type, OEMs must implement the corresponding Extender classes. The Advanced Extender implementation files are in the advanced package.

Extender classes to implement
Night advanced/NightAdvancedExtenderImpl.java
HDR advanced/HdrAdvancedExtenderImpl.java
Auto advanced/AutoAdvancedExtenderImpl.java
Bokeh advanced/BokehAdvancedExtenderImpl.java
Face Retouch advanced/BeautyAdvancedExtenderImpl.java

We use AdvancedExtenderImpl as a placeholder in the following example. Replace it with the name of the Extender file for the extension you're implementing.

Let's see how Camera2/X invokes the extensions-interface to achieve the three app flows.

App flow 1: Check extensions availability

AdvancedAppFlow1

Figure 8. App flow 1 on Advanced Extender

First, the app checks if the given extension is supported.

App flow 2: Query information

AdvancedAppFlow2

Figure 9. App flow 2 on Advanced Extender

After calling AdvancedExtenderImpl.init() , the app can query the following the information on AdvancedExtenderImpl :

  • Estimated still capture latency: AdvancedExtenderImpl.getEstimatedCaptureLatencyRange() returns the range of the capture latency for the app to evaluate if it is appropriate to enable the extension for the current scenario.

  • Supported resolutions for preview and still capture:

    • AdvancedExtenderImpl.getSupportedPreviewOutputResolutions() returns a map of image format to the sizes list that are supported for preview surface format and size. OEMs must support at least the PRIVATE format.

    • AdvancedExtenderImpl.getSupportedCaptureOutputResolutions() returns the supported format and sizes for still capture surface. OEMs must support both JPEG and YUV_420_888 format output.

    • AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() returns the supported sizes for an extra YUV_420_888 stream for image analysis. If the image analysis YUV surface isn't supported, getSupportedYuvAnalysisResolutions() should return null or an empty list.

  • Available capture request keys/results (added in extensions-interface 1.3.0): Camera2/X invokes the following methods to retrieve the supported capture request keys and result keys from your implementation:

    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys
    • AdvancedExtenderImpl.getAvailableCaptureResultKeys

For more information, see Support capture request keys and results .

App flow 3: Preview/still capture with extension enabled

AdvancedAppFlow3

Figure 10. App flow 3 on Advanced Extender

The above diagram shows the main flow for starting preview and still capture for the Advanced Extender type. Let's walk through each step.

  1. SessionProcessorImpl instance

    The core Advanced Extender implementation is in SessionProcessorImpl , which is responsible for providing customized session configuration and sending capture requests to initiate the preview and still capture request. AdvancedExtenderImpl.createSessionProcessor() is invoked to return the SessionProcessorImpl instance.

  2. initSession

    SessionProcessorImpl.initSession() initializes the session for the extension. This is where you allocate resources and return a session configuration for preparing a CameraCaptureSession .

    For the input parameters, Camera2/X specifies the output surface configurations for preview, still capture, and an optional YUV image analysis. This output surface configuration ( OutputSurfaceImpl ) contains the surface, size and image format that are retrieved by following methods in AdvancedExtenderImpl :

    • getSupportedPreviewOutputResolutions()
    • getSupportedCaptureOutputResolutions()
    • getSupportedYuvAnalysisResolutions()

    You must return a Camera2SessionConfigImpl instance, which consists of a list of Camera2OutputConfigImpl instances and the session parameters used for configuring CameraCaptureSession . You're responsible for outputting the correct camera images to the output surfaces passed in by Camera2/X. Here are some options to enable the output:

    • Processing in camera HAL: You can directly add the output surfaces to CameraCaptureSession with a SurfaceOutputConfigImpl implementation. This configures the supplied output surface to the camera pipeline and allows the camera HAL to process the image.

    • Processing intermediate ImageReader surface (RAW, YUV, etc): Add the intermediate ImageReader surfaces to the CameraCaptureSession with an ImageReaderOutputConfigImpl instance.

      You need to process the intermediate images and write the result image to the output surface.

    • Use Camera2 surface sharing: Use surface sharing with another surface by adding any Camera2OutputConfigImpl instance to the getSurfaceSharingOutputConfigs() method of another Camera2OutputConfigImpl instance. The surface format and size must be identical.

    All Camera2OutputConfigImpl including SurfaceOutputConfigImpl and ImageReaderOutputConfigImpl must have a unique ID ( getId() ), which is used to specify the target surface and retrieve the image from ImageReaderOutputConfigImpl .

  3. onCaptureSessionStart and RequestProcessorImpl

    When CameraCaptureSession starts and the Camera framework invokes onConfigured() , then Camera2/X invokes SessionProcessorImpl.onCaptureSessionStart() with the Camera2 request wrapper RequestProcessImpl . Camera2/X implements RequestProcessImpl , which enables you to execute the capture requests , and retrieve images if ImageReaderOutputConfigImpl is used.

    The RequestProcessImpl APIs are similar to the Camera2 CameraCaptureSession APIs in terms of executing requests. The differences are:

    • The target surface is specified by the ID of the Camera2OutputConfigImpl instance.
    • The capability of retrieving the image of the ImageReader .

    You can call RequestProcessorImpl.setImageProcessor() with a specified Camera2OutputConfigImpl ID to register an ImageProcessorImpl instance to receive images.

    The RequestProcessImpl instance becomes invalid after Camera2/X calls SessionProcessorImpl.onCaptureSessionEnd() .

  4. Start the preview and take a picture

    In the Advanced Extender implementation, you can send capture requests through the RequestProcessorImpl interface. Camera2/X notifies you to start the repeating request for preview or the still capture sequence by calling SessionProcessorImpl#startRepeating and SessionProcessorImpl#startCapture respectively. You should send capture requests to satisfy these preview and still-capture requests.

    Camera2/X also sets the capture request parameters through SessionProcessorImpl#setParameters . You must set these request parameters (if parameters are supported) on both the repeating and single requests.

    You must support at least CaptureRequest.JPEG_ORIENTATION and CaptureRequest.JPEG_QUALITY . extensions-interface 1.3.0 supports request and result keys, which are exposed by the following methods:

    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys()
    • AdvancedExtenderImpl.getAvailableCaptureResultKeys()

    When developers set the keys in the getAvailableCaptureRequestKeys list, you must enable the parameters and ensure the capture result contains the keys in the getAvailableCaptureResultKeys list.

  5. startTrigger

    SessionProcessorImpl.startTrigger() is invoked to start the trigger such as CaptureRequest.CONTROL_AF_TRIGGER and CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER . You can disregard any capture request keys that weren't advertised in AdvancedExtenderImpl.getAvailableCaptureRequestKeys() .

    startTrigger() has been supported since extensions-interface 1.3.0. It enables apps to implement tap-to-focus and flash with extensions.

  6. Clean up

    When finishing a capture session, SessionProcessorImpl.onCaptureSessionEnd() is invoked ahead of closing CameraCaptureSession . After the capture session has closed, deInitSession() performs the clean up.

Support preview, still capture, and image analysis

You should apply the extension for both the preview and still capture use cases. However, if the latency is too high to smoothly show the preview, you can apply the extension only for still capture.

For the Basic Extender type, regardless of enabling the extension for preview, you must implement both ImageCaptureExtenderImpl and PreviewExtenderImpl for a given extension. Often, an app also uses a YUV stream to analyze the image content such as finding QR codes or text. To better support this use case , you should support the stream combination of preview, still capture, and a YUV_420_888 stream for configuring CameraCaptureSession . This means that if you implement a processor, then you have to support the stream combination of three YUV_420_888 streams.

For Advanced Extender, Camera2/X passes three output surfaces to the SessionProcessorImpl.initSession() call. These output surfaces are for preview , still capture, and image analysis, respectively. You must ensure that preview and still capture output surfaces show the valid output. However, for the image analysis output surface, ensure it's working only when it's non-null. If your implementation can't support the image analysis stream, you can return an empty list in AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() . This ensures the image analysis output surface is always null in SessionProcessorImpl.initSession() .

Support video capture

The current Camera Extension architecture supports only the preview and still capture use cases. We don't support enabling the extension on the MediaCodec or MediaRecorder surfaces for recording the video. However, it's possible for apps to record the preview output.

Supporting MediaCodec and MediaRecorder surfaces is under investigation.

Extensions interface version history

The following table shows the Camera Extension interface version history. You should always implement the vendor library with the latest version.

Version Added features
1.0.0
  • Version verification
    • ExtensionVersionImpl
  • Basic Extender
    • PreviewExtenderImpl
    • ImageCaptureExtenderImpl
    • Processor
      • PreviewImageProcessorImpl
      • CaptureProcessorImpl
      • RequestUpdateProcessorImpl
1.1.0
  • Library initialization
    • InitializerImpl
  • Expose supported resolutions
    • PreviewExtenderImpl.getSupportedResolutions
    • ImageCaptureExtenderImpl.getSupportedResolutions
1.2.0
  • AdvancedExtender
    • AdvancedExtenderImpl
    • SessionProcessorImpl
  • Get estimated capture latency
    • ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange
1.3.0
  • Expose supported capture request keys/results keys
    • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys
    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys
    • New process() call that takes ProcessResultImpl in PreviewImageProcessorImpl and CaptureProcessorImpl
    • Support trigger type request
      • AdvancedExtenderImpl.startTrigger

Reference implementation

The following reference OEM vendor library implementations are available in frameworks/ex .

  • advancedSample : A basic implementation of Advanced Extender.

  • sample : A basic implementation of Basic Extender.

  • service_based_sample : An implementation that demonstrates how to host Camera Extensions in a Service . This implementation contains the following components:

    • oem_library : A Camera Extensions OEM library for Camera2 and CameraX Extensions APIs that implements Extensions-Interface . This acts as a passthrough that forwards calls from Extensions-Interface to the service. This library also provides AIDL files and wrapper classes to communicate with the service.

      Advanced Extender is enabled by default. To enable the Basic Extender, change ExtensionsVersionImpl#isAdvancedExtenderImplemented to return false .

    • extensions_service : A sample implementation of the Extensions Service. Add your implementation here. The interface to implement in the service is similar to the Extensions-Interface . For example, implementing the IAdvancedExtenderImpl.Stub performs the same operations as AdvancedExtenderImpl . ImageWrapper and TotalCaptureResultWrapper are required to make Image and TotalCaptureResult parcelable.

Set up the vendor library on a device

The OEM vendor library isn't built into an app; it's loaded from the device at runtime by Camera2/X. In CameraX, the <uses-library> tag declares that the androidx.camera.extensions.impl library, which is defined in the AndroidManifest.xml file of the camera-extensions library, is a dependency of CameraX and must be loaded at runtime. In Camera2, the framework loads an extensions service that also declares that the <uses-library> loads the same androidx.camera.extensions.impl library at runtime.

This allows third-party apps using extensions to automatically load the OEM vendor library. The OEM library is marked as optional so apps can run on devices that don't have the library on the device. Camera2/X handles this behavior automatically when an app tries to use a camera extension as long as the device manufacturer places the OEM library on the device so that it can be discovered by the app.

To set up the OEM library on a device, do the following:

  1. Add a permission file, which is required by the <uses-library> tag, using the following format: /etc/permissions/ ANY_FILENAME .xml . For example, /etc/permissions/camera_extensions.xml . The files in this directory provide a mapping of the library named in <uses-library> to the actual file path on the device.

  2. Use the example below to add the required information to the file.

    • name must be androidx.camera.extensions.impl as that's the library that CameraX searches for.
    • file is the absolute path of the file that contains the extensions implementation (for example, /system/framework/androidx.camera.extensions.impl.jar ).
    <?xml version="1.0" encoding="utf-8"?>
<permissions>
    <library name="androidx.camera.extensions.impl"
             file="OEM_IMPLEMENTED_JAR" />
</permissions>

In Android 12 or higher, devices supporting CameraX extensions must have the ro.camerax.extensions.enabled property set to true , which allows for querying whether a device supports extensions. To do this, add the following line in the device make file:

PRODUCT_VENDOR_PROPERTIES += \
    ro.camerax.extensions.enabled=true \

Validation

To test your implementation of the OEM vendor library during the development stage, use the example app at androidx-main/camera/integration-tests/extensionstestapp/ , which runs through various vendor extensions.

After you complete your implementation, use the Camera Extensions Validation Tool to run automated and manual tests to verify that the vendor library is implemented correctly.

Extended scene mode versus Camera Extensions

For the bokeh extension, in addition to exposing it using Camera Extensions, you can expose the extension using the extended scene mode, which is enabled through the CONTROL_EXTENDED_SCENE_MODE key. For more implementation details, see Camera Bokeh .

Extended scene mode has fewer restrictions compared to Camera Extensions for camera2 apps. For example, you can enable extended scene mode in a regular CameraCaptureSession instance that supports flexible stream combinations and capture request parameters. In contrast, camera extensions support only a fixed set of stream types and have limited support for capture request parameters.

A downside of extended scene mode is that you can only implement it in the camera HAL, which means that it must be verified to work across all orthogonal controls available to app developers.

We recommend exposing bokeh using both the extended scene mode and Camera Extensions because apps might prefer to use a particular API to enable bokeh. We recommend first using the extended scene mode because this is the most flexible way for apps to enable the bokeh extension. Then you can implement the camera extensions interface based on the extended scene mode. If implementing bokeh in the camera HAL is difficult, for example, because it requires a post processor running in the app layer to process images, we recommend implementing the bokeh extension using the Camera Extensions interface.

Frequently asked questions (FAQs)

Are there any restrictions on API levels?

Yes. This depends on the Android API feature set that's required by the OEM vendor library implementation. For example, ExtenderStateListener.onPresetSession() uses the SessionConfiguration.setSessionParameters() call to set a baseline set of tags. This call is available only on API level 28 and higher. For details on specific interface methods, see the API reference documentation .