Extensions de caméra

Les fabricants d'appareils peuvent exposer des extensions telles que le bokeh, le mode nuit et le HDR à des développeurs tiers via l'interface Camera Extensions fournie par la bibliothèque du fournisseur OEM. Les développeurs peuvent utiliser l' API Camera2 Extensions et l' API CameraX Extensions pour accéder aux extensions implémentées dans la bibliothèque du fournisseur OEM.

Pour une liste des extensions prises en charge, qui est la même pour Camera2 et CameraX, voir API CameraX Extensions . Si vous souhaitez ajouter une extension, signalez un bogue avec l' Issue Tracker .

Cette page décrit comment implémenter et activer la bibliothèque du fournisseur OEM sur les appareils.

Architecture

Le schéma suivant décrit l'architecture de l'interface Camera Extensions ou extensions-interface : Architecture

Figure 1. Schéma d'architecture des extensions de caméra

Comme indiqué dans le diagramme, pour prendre en charge les extensions de caméra, vous devez implémenter l' extensions-interface fournie par la bibliothèque du fournisseur OEM. Votre bibliothèque de fournisseur OEM active deux API : l'API d'extensions CameraX et l'API d'extensions Camera2 , qui sont utilisées respectivement par les applications CameraX et Camera2 pour accéder aux extensions de fournisseur.

Implémenter la bibliothèque de fournisseurs OEM

Pour implémenter la bibliothèque du fournisseur OEM, copiez les fichiers camera-extensions-stub dans un projet de bibliothèque système. Ces fichiers définissent l'interface Camera Extensions.

Les fichiers camera-extensions-stub sont divisés dans les catégories suivantes :

Fichiers d'interface essentiels (ne pas modifier)

  • PreviewExtenderImpl.java
  • ImageCaptureExtenderImpl.java
  • ExtenderStateListener.java
  • ProcessorImpl.java
  • PreviewImageProcessorImpl.java
  • CaptureProcessorImpl.java
  • CaptureStageImpl.java
  • RequestUpdateProcessorImpl.java
  • ProcessResultImpl.java
  • advanced/AdvancedExtenderImpl.java
  • advanced/Camera2OutputConfigImpl.java
  • advanced/Camera2SessionConfigImpl.java
  • advanced/ImageProcessorImpl.java
  • advanced/ImageReaderOutputConfigImpl.java
  • advanced/ImageReferenceImpl.java
  • advanced/MultiResolutionImageReaderOutputConfigImpl.java
  • advanced/OutputSurfaceImpl.java
  • advanced/RequestProcessorImpl.java
  • advanced/SessionProcessorImpl.java
  • advanced/SurfaceOutputConfigImpl.java

Implémentations obligatoires (ajoutez votre implémentation)

  • ExtensionVersionImpl.java
  • InitializerImpl.java

Classes d'extension Bokeh (implémentez-les si l'extension Bokeh est prise en charge)

  • BokehImageCaptureExtenderImpl.java
  • BokehPreviewExtenderImpl.java
  • advanced/BokehAdvancedExtenderImpl.java

Classes d'extension de nuit (implémentez-les si l'extension de nuit est prise en charge)

  • NightImageCaptureExtenderImpl.java
  • NightPreviewExtenderImpl.java
  • advanced/NightAdvancedExtenderImpl.java

Classes d'extension automatique (implémentez-les si l'extension automatique est prise en charge)

  • AutoImageCaptureExtenderImpl.java
  • AutoPreviewExtenderImpl.java
  • advanced/AutoAdvancedExtenderImpl.java

Classes d'extension HDR (implémentez-les si l'extension HDR est prise en charge)

  • HdrImageCaptureExtenderImpl.java
  • HdrPreviewExtenderImpl.java
  • advanced/HdrAdvancedExtenderImpl.java

Classes d'extension Face Retouch (implémentez-les si l'extension Face Retouch est prise en charge)

  • BeautyImageCaptureExtenderImpl.java
  • BeautyPreviewExtenderImpl.java
  • advanced/BeautyAdvancedExtenderImpl.java

Utilitaires (facultatif, peut être supprimé)

  • advanced/Camera2OutputConfigImplBuilder.java
  • advanced/Camera2SessionConfigImplBuilder.java

Vous n'êtes pas obligé de fournir une implémentation pour chaque extension. Si vous n'implémentez pas d'extension, définissez isExtensionAvailable() pour renvoyer false ou supprimez les classes Extender correspondantes. Les API Camera2 et CameraX Extensions signalent à l'application que l'extension n'est pas disponible.

Voyons comment les API Camera2 et CameraX Extensions interagissent avec la bibliothèque du fournisseur pour activer une extension. Le schéma suivant illustre le flux de bout en bout en utilisant l'extension Nuit comme exemple :

Flux principal

Figure 2. Mise en œuvre de l'extension de nuit

  1. Vérification de version :

    Camera2/X appelle ExtensionVersionImpl.checkApiVersion() pour s'assurer que la version extensions-interface implémentée par l'OEM est compatible avec les versions prises en charge par Camera2/X.

  2. Initialisation de la bibliothèque fournisseur :

    InitializerImpl a une méthode init() qui initialise la bibliothèque du fournisseur. Camera2/X termine l'initialisation avant d'accéder aux classes Extender.

  3. Instancier les classes Extender :

    Instancie les classes Extender pour l'extension. Il existe deux types d'extendeur : l'extendeur de base et l'extendeur avancé. Vous devez implémenter un type d'extendeur pour toutes les extensions. Pour plus d'informations, consultez Basic Extender versus Advanced Extender .

    Camera2/X instancie et interagit avec les classes Extender pour récupérer des informations et activer l'extension. Pour une extension donnée, Camera2/X peut instancier plusieurs fois les classes Extender. Par conséquent, n'effectuez pas d'initialisation lourde dans le constructeur ou l'appel init() . Ne faites le gros du travail que lorsque la session de la caméra est sur le point de démarrer, par exemple lorsque onInit() est appelé dans Basic Extender ou initSession() est appelé dans Advanced Extender.

    Pour l'extension Night, les classes Extender suivantes sont instanciées pour le type Basic Extender :

    • NightImageCaptureExtenderImpl.java
    • NightPreviewExtenderImpl.java

    Et pour le type Advanced Extender :

    • NightAdvancedExtenderImpl.java

  4. Vérifier la disponibilité de l'extension :

    Avant d'activer l'extension, isExtensionAvailable() vérifie si l'extension est disponible sur l'ID de caméra spécifié via l'instance Extender.

  5. Initialisez l'extendeur avec les informations de la caméra :

    Camera2/X appelle init() sur l'instance Extender et lui transmet l'ID de la caméra et CameraCharacteristics .

  6. Informations sur la requête :

    Appelle la classe Extender pour récupérer des informations telles que les résolutions prises en charge, toujours capturer la latence estimée et capturer les clés de demande de l'extendeur en vue de l'activation de l'extension.

  7. Activer l'extension sur l'extendeur :

    La classe Extender fournit toutes les interfaces nécessaires pour activer la classe. Il offre un mécanisme pour accrocher l'implémentation OEM dans le pipeline Camera2, comme l'injection de paramètres de demande de capture ou l'activation d'un post-processeur.

    Pour le type Advanced Extender, Camera2/X interagit avec SessionProcessorImpl pour activer l'extension. Camera2/X récupère l'instance SessionProcessorImpl en appelant createSessionProcessor() sur l'extendeur.

Les sections suivantes décrivent le flux d'extension plus en détail.

Vérification de version

Lors du chargement de la bibliothèque du fournisseur OEM à partir de l'appareil lors de l'exécution, Camera2/X vérifie si la bibliothèque est compatible avec la version extensions-interface . L' extensions-interface utilise le versioning sémantique, ou MAJOR.MINOR.PATCH, par exemple, 1.1.0 ou 1.2.0. Cependant, seules les versions majeures et mineures sont utilisées lors de la vérification de version.

Pour vérifier la version, Camera2/X appelle ExtensionVersionImpl.checkApiVersion() avec la version extensions-interface prise en charge. Camera2/X utilise ensuite la version signalée par la bibliothèque OEM pour déterminer si l'extension peut être activée et quelles fonctionnalités elle doit invoquer.

Compatibilité des versions majeures

Si les principales versions de l' interface d'extension sont différentes entre Camera2/X et la bibliothèque du fournisseur, elle est alors considérée comme incompatible et l'extension est désactivée.

Rétrocompatibilité

Tant que la version majeure est identique, Camera2/X assure la rétrocompatibilité avec les bibliothèques de fournisseurs OEM construites avec des versions extensions-interface antérieures. Par exemple, si Camera2/X prend en charge extensions-interface 1.3.0, les bibliothèques du fournisseur OEM qui ont implémenté 1.0.0, 1.1.0 et 1.2.0 sont toujours compatibles. Cela signifie également qu'après avoir implémenté une version spécifique de la bibliothèque du fournisseur, Camera2/X s'assure que la bibliothèque est rétrocompatible avec les prochaines versions extension-interface .

Compatibilité ascendante

La compatibilité ascendante avec les bibliothèques de fournisseurs des extensions-interface les plus récentes dépend de vous, l'OEM. Si vous avez besoin de certaines fonctionnalités pour implémenter les extensions, vous pouvez activer les extensions à partir d'une certaine version. Dans ce cas, vous pouvez renvoyer la version extensions-interface prise en charge lorsque la version de la bibliothèque Camera2/X répond aux exigences. Si les versions Camera2/X ne sont pas prises en charge, vous pouvez renvoyer une version incompatible telle que 99.0.0 pour désactiver les extensions.

Initialisation de la bibliothèque du fournisseur

Après avoir vérifié la version extensions-interface implémentée par la bibliothèque OEM, Camera2/X démarre le processus d'initialisation. La méthode InitializerImpl.init() signale à la bibliothèque OEM qu'une application tente d'utiliser des extensions.

Camera2/X n'effectue aucun autre appel à la bibliothèque OEM (hormis la vérification de version) jusqu'à ce que la bibliothèque du fournisseur OEM appelle OnExtensionsInitializedCallback.onSuccess() pour notifier la fin de l'initialisation.

Vous devez implémenter InitializerImpl à partir d' extensions-interface 1.1.0. Camera2/X ignore l'étape d'initialisation de la bibliothèque si la bibliothèque du fournisseur OEM implémente extensions-interface 1.0.0.

Prolongateur de base contre prolongateur avancé

Il existe deux types d'implémentation extensions-interface : Basic Extender et Advanced Extender. Advanced Extender est pris en charge depuis extensions-interface 1.2.0.

Implémentez Basic Extender pour les extensions qui traitent les images dans la couche HAL de la caméra ou utilisez un post-processeur capable de traiter les flux YUV.

Implémentez Advanced Extender pour les extensions qui doivent personnaliser la configuration du flux Camera2 et envoyer des demandes de capture si nécessaire.

Voir le tableau suivant pour la comparaison :

Prolongateur de base Prolongateur avancé
Configurations de flux Fixé
Aperçu : PRIVATE ou YUV_420_888 (si le processeur existe)
Capture fixe : JPEG ou YUV_420_888 (si le processeur existe)		 Personnalisable par OEM.
Envoi de la demande de capture Seul Camera2/X peut envoyer des demandes de capture. Vous pouvez paramétrer ces requêtes. Lorsque le processeur est fourni pour la capture d'image, Camera2/X peut envoyer plusieurs demandes de capture et envoyer toutes les images et les résultats de capture au processeur. Une instance RequestProcessorImpl vous est fournie pour exécuter la requête de capture camera2 et obtenir les résultats et Image.

Camera2/X invoque startRepeating et startCapture sur SessionProcessorImpl pour signaler à l'OEM de démarrer la demande répétée d'aperçu et de démarrer la séquence de capture fixe respectivement.

Crochets dans le pipeline de la caméra
  • onPresetSession fournit des paramètres de session.
  • onEnableSession envoie une seule requête juste après la configuration CameraCaptureSession .
  • onDisableSession envoie une seule requête avant que CameraCaptureSession ne soit fermée.
  • initSession initialise et renvoie une configuration de session camera2 personnalisée pour créer la session de capture.
  • onCaptureSessionStart est appelé juste après la configuration CameraCaptureSession .
  • onCaptureSessionEnd est appelé avant que CameraCaptureSession ne soit fermée.
Convient à Extensions implémentées dans la caméra HAL ou dans un processeur qui traite les images YUV.
  • Possède des implémentations basées sur Camera2 pour les extensions.
  • Nécessite une configuration de flux personnalisée telle que le flux RAW.
  • Nécessite une séquence de capture interactive.
Version d'API prise en charge Extensions Camera2 : Android 13 ou supérieur
Extensions CameraX : camera-extensions 1.1.0 ou supérieures		 Extensions Camera2 : Android 12L ou supérieur
Extensions CameraX : camera-extensions 1.2.0-alpha03 ou supérieur

Flux d'application

Le tableau suivant présente trois types de flux d'application et leurs appels d'API Camera Extensions correspondants. Bien que Camera2/X fournisse ces API, vous devez implémenter correctement la bibliothèque du fournisseur pour prendre en charge ces flux, que nous décrivons plus en détail dans une section ultérieure.

Extensions caméra2 Extensions CameraX
Disponibilité des extensions de requête CameraExtensionCharacteristics . getSupportedExtensions ExtensionsManager. isExtensionAvailable
Informations sur la requête CameraExtensionCharacteristics. getExtensionSupportedSizes CameraExtensionCharacteristics. getEstimatedCaptureLatencyRangeMillis CameraExtensionCharacteristics. getAvailableCaptureRequestKeys CameraExtensionCharacteristics. getAvailableCaptureResultKeys ExtensionsManager. getEstimatedCaptureLatencyRange

CameraX gère le reste des informations dans la bibliothèque.

Aperçu et capture fixe avec extension activée CameraDevice. createExtensionSession

cameraExtensionsSession. setRepeatingRequest

cameraExtensionsSession. capture

 val cameraSelector = ExtensionsManager. getExtensionEnabledCameraSelector

bindToLifecycle(lifecycleOwner, cameraSelector, aperçu, ...)

Prolongateur de base

L'interface Basic Extender fournit des crochets à plusieurs endroits dans le pipeline de la caméra. Chaque type d'extension a des classes Extender correspondantes que les OEM doivent implémenter.

Le tableau suivant répertorie les classes d'extension que les OEM doivent implémenter pour chaque extension :

Classes d'extension à implémenter
Nuit NightPreviewExtenderImpl.java

NightImageCaptureExtenderImpl.java

HDR HdrPreviewExtenderImpl.java

HdrImageCaptureExtenderImpl.java

Auto AutoPreviewExtenderImpl.java

AutoImageCaptureExtenderImpl.java

Bokeh BokehPreviewExtenderImpl.java

BokehImageCaptureExtenderImpl.java

Retouche visage BeautyPreviewExtenderImpl.java

BeautyImageCaptureExtenderImpl.java

Nous utilisons PreviewExtenderImpl et ImageCaptureExtenderImpl comme espaces réservés dans l'exemple suivant. Remplacez-les par les noms des fichiers réels que vous implémentez.

Basic Extender a les fonctionnalités suivantes :

  • Injectez les paramètres de session lors de la configuration CameraCaptureSession ( onPresetSession ).
  • Vous avertir des événements de démarrage et de fermeture de la session de capture et envoyer une seule requête pour notifier HAL avec les paramètres renvoyés ( onEnableSession , onDisableSession ).
  • Injectez les paramètres de capture pour la requête ( PreviewExtenderImpl.getCaptureStage , ImageCaptureExtenderImpl.getCaptureStages ).
  • Ajoutez des processeurs pour la prévisualisation et la capture continue capables de traiter le flux YUV_420_888 .

Voyons comment Camera2/X invoque l' extensions-interface pour réaliser les trois flux d'application mentionnés ci-dessus.

Flux d'application 1 : Vérifier la disponibilité de l'extension

BasicExtenderAppFlow1

Figure 3. Flux d'application 1 sur Basic Extender

Dans ce flux, Camera2/X appelle directement la méthode isExtensionAvailable() de PreviewExtenderImpl et ImageCaptureExtenderImpl sans appeler init() . Les deux classes Extender doivent renvoyer true pour activer les extensions.

Il s'agit souvent de la première étape pour les applications de vérifier si le type d'extension donné est pris en charge pour un ID de caméra donné avant d'activer l'extension. En effet, certaines extensions ne sont prises en charge que sur certains ID de caméra.

Flux d'application 2 : informations sur les requêtes

BasicExtenderAppFlow2

Figure 4. Flux d'application 2 sur Basic Extender

Après avoir déterminé si l'extension est disponible, les applications doivent demander les informations suivantes avant d'activer l'extension.

  • Toujours capturer la plage de latence : ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange renvoie la plage de latence de capture pour que l'application évalue s'il est approprié d'activer l'extension pour le scénario actuel.

  • Tailles prises en charge pour la surface d'aperçu et de capture : ImageCaptureExtenderImpl.getSupportedResolutions et PreviewExtenderImpl.getSupportedResolutions renvoient une liste des formats d'image et des tailles prises en charge pour le format et la taille de la surface.

  • Clés de demande et de résultat prises en charge : Camera2/X invoque les méthodes suivantes pour récupérer les clés de demande de capture et les clés de résultat prises en charge à partir de votre implémentation :

    • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys
    • ImageCaptureExtenderImpl.getAvailableCapturetResultKeys

Camera2/X appelle toujours init() en premier sur ces classes Extender avant de demander plus d'informations.

Flux d'application 3 : Aperçu/capture fixe avec l'extension activée (mise en œuvre HAL)

BasicExtenderAppFlow3

Figure 5. Flux d'application 3 sur Basic Extender

Le diagramme ci-dessus illustre le flux principal d'activation de la prévisualisation et de la capture fixe avec une extension sans processeur. Cela signifie que la caméra HAL traite l'extension.

Dans ce flux, Camera2/X appelle d'abord init() puis onInit , qui vous avertit qu'une session de caméra est sur le point de démarrer avec les extensions spécifiées. Vous pouvez effectuer une initialisation lourde dans onInit() .

Lors de la configuration CameraCaptureSession , Camera2/X invoque onPresetSession pour obtenir les paramètres de session. Une fois la session de capture configurée avec succès, Camera2/X appelle onEnableSession en renvoyant une instance CaptureStageImpl qui contient les paramètres de capture. Camera2/X envoie immédiatement une requête unique avec ces paramètres de capture pour notifier le HAL. De même, avant la fermeture de la session de capture, Camera2/X invoque onDisableSession puis envoie une seule requête avec les paramètres de capture renvoyés.

La demande répétée déclenchée par Camera2/X contient les paramètres de demande renvoyés par PreviewExtenderImpl.getCaptureStage() . De plus, la demande de capture fixe contient les paramètres renvoyés par ImageCaptureExtenderImpl.getCaptureStages() .

Enfin, Camera2/X invoque onDeInit() une fois la session de caméra terminée. Vous pouvez libérer des ressources dans onDeinit() .

Processeur de prévisualisation

En plus de la caméra HAL, vous pouvez également implémenter des extensions dans un processeur.

Implémentez PreviewExtenderImpl.getProcessorType pour spécifier le type de processeur comme expliqué ci-dessous :

  • PROCESSOR_TYPE_NONE : aucun processeur. Les images sont traitées dans la caméra HAL.

  • PROCESSOR_TYPE_REQUEST_UPDATE_ONLY : le type de processeur vous permet de mettre à jour la demande répétée avec de nouveaux paramètres de demande de capture basés sur le dernier TotalCaptureResult .

    PreviewExtenderImpl.getProcessor doit renvoyer une instance RequestUpdateProcessorImpl qui traite l'instance TotalCaptureResult et renvoie une instance CaptureStageImpl pour mettre à jour la demande répétée. PreviewExtenderImpl.getCaptureStage() doit également refléter le résultat du traitement et renvoyer le dernier CaptureStageImpl .

  • PROCESSOR_TYPE_IMAGE_PROCESSOR : ce type vous permet d'implémenter un processeur pour traiter les images YUV_420_888 et écrire la sortie sur une surface PRIVATE .

    Vous devez implémenter et renvoyer une instance PreviewImageProcessorImpl dans PreviewExtenderImpl.getProcessor . Le processeur est responsable du traitement des images d'entrée YUV_420_888 . Il doit écrire la sortie au format PRIVATE de l'aperçu. Camera2/X utilise une surface YUV_420_888 au lieu de PRIVATE pour configurer CameraCaptureSession pour l'aperçu.

    Voir l'illustration suivante pour le flux :

AperçuProcesseur

Figure 6. Flux de prévisualisation avec PreviewImageProcessorImpl

L'interface PreviewImageProcessorImpl étend ProcessImpl et possède trois méthodes importantes :

  • onOutputSurface(Surface surface, int imageFormat) définit la surface de sortie pour le processeur. Pour PreviewImageProcessorImpl , imageFormat est un format de pixel tel que PixelFormat.RGBA_8888 .

  • onResolutionUpdate(Size size) définit la taille de l'image d'entrée.

  • onImageFormatUpdate(int imageFormat) définit le format d'image de l'image d'entrée. Actuellement, il ne peut s'agir que YUV_420_888 .

Processeur de capture d'images

Pour la capture fixe, vous pouvez implémenter un processeur en renvoyant une instance CaptureProcessorImpl à l'aide de ImageCaptureExtenderImpl.getCaptureProcessor . Le processeur est chargé de traiter une liste d'images YUV_420_888 capturées et d'instances TotalCaptureResult et d'écrire la sortie sur une surface YUV_420_888 .

Vous pouvez supposer en toute sécurité que l'aperçu est activé et en cours d'exécution avant d'envoyer la demande de capture d'image fixe.

Voir le flux dans le diagramme ci-dessous :

CaptureProcesseur

Figure 7. Toujours capturer le flux avec CaptureProcessorImpl

  1. Camera2/X utilise une surface au format YUV_420_888 pour la capture fixe afin de configurer la session de capture. Camera2/X prépare CaptureProcessorImpl en appelant :

    • CaptureProcessorImpl.onImageFormatUpdate() avec YUV_420_888 .
    • CaptureProcessorImpl.onResolutionUpdate() avec la taille de l'image d'entrée.
    • CaptureProcessorImpl.onOutputSurface() avec une surface de sortie YUV_420_888 .

  2. ImageCaptureExtenderImpl.getCaptureStages renvoie une liste de CaptureStageImpl , où chaque élément correspond à une instance CaptureRequest avec des paramètres de capture envoyés par Camera2/X. Par exemple, s'il renvoie une liste de trois instances CaptureStageImpl , Camera2/X envoie trois demandes de capture avec les paramètres de capture correspondants à l'aide de l'API captureBurst .

  3. Les images reçues et les instances TotalCaptureResult sont regroupées et envoyées à CaptureProcessorImpl pour traitement.

  4. CaptureProcessorImpl écrit le résultat Image (format YUV_420_888 ) sur la surface de sortie spécifiée par l'appel onOutputSurface() . Camera2/X le convertit en images JPEG si nécessaire.

Prend en charge les clés et les résultats des demandes de capture

En plus de l'aperçu et de la capture de l'appareil photo, les applications peuvent définir le zoom, les paramètres du flash ou déclencher une mise au point par pression. Ces paramètres peuvent ne pas être compatibles avec l'implémentation de votre extension.

Les méthodes suivantes ont été ajoutées à extensions-interface 1.3.0 pour vous permettre d'exposer les paramètres pris en charge par votre implémentation :

  • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys() renvoie les clés de demande de capture prises en charge par votre implémentation.
  • ImageCaptureExtenderImpl.getAvailableCaptureResultKeys() renvoie les clés de résultat de capture contenues dans le résultat de capture.

Si la caméra HAL traite l'extension, Camera2/X récupère les résultats de la capture dans CameraCaptureSession.CaptureCallback . Cependant, si le processeur est implémenté, Camera2/X récupère les résultats de la capture dans ProcessResultImpl , qui est transmis à la méthode process() dans PreviewImageProcessorImpl et CaptureProcessorImpl . Vous êtes chargé de signaler le résultat de la capture via ProcessResultImpl à Camera2/X.

Voir la définition de l'interface CaptureProcessorImpl ci-dessous à titre d'exemple. Dans extensions-interface 1.3.0 ou supérieur, le deuxième appel process() est invoqué :

Interface CaptureProcessorImpl extends ProcessorImpl {
    // invoked when extensions-interface version < 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results);
    // invoked when extensions-interface version >= 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results,
            ProcessResultImpl resultCallback, Executor executor);
}

Pour les opérations courantes de l'appareil photo telles que le zoom, la mise au point par pression, le flash et la compensation d'exposition, nous vous recommandons de prendre en charge les touches suivantes pour la demande de capture et le résultat de la capture :

  • Zoom:
    • CaptureRequest#CONTROL_ZOOM_RATIO
    • CaptureRequest#SCALER_CROP_REGION
  • Appuyez pour faire la mise au point :
    • CaptureRequest#CONTROL_AF_MODE
    • CaptureRequest#CONTROL_AF_TRIGGER
    • CaptureRequest#CONTROL_AF_REGIONS
    • CaptureRequest#CONTROL_AE_REGIONS
    • CaptureRequest#CONTROL_AWB_REGIONS
  • Éclair:
    • CaptureRequest#CONTROL_AE_MODE
    • CaptureRequest#CONTROL_AE_PRECAPTURE_TRIGGER
    • CaptureRequest#FLASH_MODE
  • La compensation d'exposition:
    • CaptureRequest#CONTROL_AE_EXPOSURE_COMPENSATION

Pour les prolongateurs de base qui implémentent 1.2.0 ou des versions antérieures, l'API CameraX Extensions prend explicitement en charge toutes les clés ci-dessus. Pour extensions-interface 1.3.0, CameraX et Camera2 respectent la liste renvoyée et ne prennent en charge que les clés qu'elle contient. Par exemple, si vous décidez de renvoyer uniquement CaptureRequest#CONTROL_ZOOM_RATIO et CaptureRequest#SCALER_CROP_REGION dans l'implémentation 1.3.0, cela signifie que seul le zoom est pris en charge pour l'application tandis que la mise au point, le flash et la compensation d'exposition ne sont pas autorisés.

Prolongateur avancé

Advanced Extender est un type d'implémentation de fournisseur basé sur l'API Camera2. Ce type d'extension a été ajouté dans extensions-interface 1.2.0. Selon le fabricant de l'appareil, des extensions peuvent être implémentées dans la couche d'application, qui dépend des facteurs suivants :

  • Configuration de flux personnalisée : configurez des flux personnalisés tels que le flux RAW ou disposez de plusieurs flux pour différents ID de caméra physique.

  • Capacité d'envoyer des requêtes Camera2 : Prend en charge une logique d'interaction compliquée qui peut envoyer des requêtes de capture avec des paramètres basés sur les résultats des requêtes précédentes.

Advanced Extender fournit un wrapper, ou une couche intermédiaire, afin que vous puissiez personnaliser la configuration du flux et envoyer des demandes de capture à la demande.

Fichiers à implémenter

Pour passer à l'implémentation d'Advanced Extender, la méthode isAdvancedExtenderImplemented() dans ExtensionVersionImpl doit renvoyer true . Pour chaque type d'extension, les OEM doivent implémenter les classes Extender correspondantes. Les fichiers d'implémentation d'Advanced Extender se trouvent dans le package avancé .

Classes d'extension à implémenter
Nuit advanced/NightAdvancedExtenderImpl.java
HDR advanced/HdrAdvancedExtenderImpl.java
Auto advanced/AutoAdvancedExtenderImpl.java
Bokeh advanced/BokehAdvancedExtenderImpl.java
Retouche du visage advanced/BeautyAdvancedExtenderImpl.java

Nous utilisons AdvancedExtenderImpl comme espace réservé dans l'exemple suivant. Remplacez-le par le nom du fichier Extender pour l'extension que vous implémentez.

Voyons comment Camera2/X invoque l' extensions-interface pour réaliser les trois flux d'application.

Flux d'application 1 : vérifier la disponibilité des extensions

AdvancedAppFlow1

Figure 8. Flux d'application 1 sur Advanced Extender

Tout d'abord, l'application vérifie si l'extension donnée est prise en charge.

Flux d'application 2 : informations sur les requêtes

AdvancedAppFlow2

Figure 9. Flux d'application 2 sur Advanced Extender

Après avoir appelé AdvancedExtenderImpl.init() , l'application peut interroger les informations suivantes sur AdvancedExtenderImpl :

  • Latence de capture encore estimée : AdvancedExtenderImpl.getEstimatedCaptureLatencyRange() renvoie la plage de latence de capture pour que l'application évalue s'il est approprié d'activer l'extension pour le scénario actuel.

  • Résolutions prises en charge pour l'aperçu et la capture fixe :

    • AdvancedExtenderImpl.getSupportedPreviewOutputResolutions() renvoie une carte du format d'image à la liste des tailles prises en charge pour le format et la taille de la surface d'aperçu. Les OEM doivent prendre en charge au moins le format PRIVATE .

    • AdvancedExtenderImpl.getSupportedCaptureOutputResolutions() renvoie le format et les tailles pris en charge pour la surface de capture fixe. Les OEM doivent prendre en charge les formats de sortie JPEG et YUV_420_888 .

    • AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() renvoie les tailles prises en charge pour un flux YUV_420_888 supplémentaire pour l'analyse d'image. Si la surface YUV d'analyse d'image n'est pas prise en charge, getSupportedYuvAnalysisResolutions() doit renvoyer null ou une liste vide.

  • Clés/résultats de demande de capture disponibles (ajoutés dans extensions-interface 1.3.0) : Camera2/X invoque les méthodes suivantes pour récupérer les clés de demande de capture et les clés de résultat prises en charge à partir de votre implémentation :

    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys
    • AdvancedExtenderImpl.getAvailableCaptureResultKeys

Pour plus d'informations, consultez Prise en charge des clés de demande de capture et des résultats .

Flux d'application 3 : Aperçu/capture fixe avec l'extension activée

AdvancedAppFlow3

Figure 10. Flux d'application 3 sur Advanced Extender

Le diagramme ci-dessus montre le flux principal pour démarrer l'aperçu et continuer à capturer pour le type Advanced Extender. Passons en revue chaque étape.

  1. Instance SessionProcessorImpl

    L'implémentation principale d'Advanced Extender se trouve dans SessionProcessorImpl , qui est chargé de fournir une configuration de session personnalisée et d'envoyer des demandes de capture pour lancer la demande d'aperçu et de capture. AdvancedExtenderImpl.createSessionProcessor() est invoqué pour renvoyer l'instance SessionProcessorImpl .

  2. initSession

    SessionProcessorImpl.initSession() initialise la session pour l'extension. C'est là que vous allouez des ressources et renvoyez une configuration de session pour préparer une CameraCaptureSession .

    Pour les paramètres d'entrée, Camera2/X spécifie les configurations de surface de sortie pour l'aperçu, la capture fixe et une analyse d'image YUV facultative. Cette configuration de surface de sortie ( OutputSurfaceImpl ) contient la surface, la taille et le format d'image qui sont récupérés par les méthodes suivantes dans AdvancedExtenderImpl :

    • getSupportedPreviewOutputResolutions()
    • getSupportedCaptureOutputResolutions()
    • getSupportedYuvAnalysisResolutions()

    Vous devez renvoyer une instance Camera2SessionConfigImpl , qui se compose d'une liste d'instances Camera2OutputConfigImpl et des paramètres de session utilisés pour configurer CameraCaptureSession . Vous êtes responsable de la sortie des images de caméra correctes sur les surfaces de sortie transmises par Camera2/X. Voici quelques options pour activer la sortie :

    • Traitement dans la caméra HAL : vous pouvez directement ajouter les surfaces de sortie à CameraCaptureSession avec une implémentation SurfaceOutputConfigImpl . Cela configure la surface de sortie fournie au pipeline de la caméra et permet à la caméra HAL de traiter l'image.

    • Traitement de la surface ImageReader intermédiaire (RAW, YUV, etc.) : Ajoutez les surfaces ImageReader intermédiaires à CameraCaptureSession avec une instance ImageReaderOutputConfigImpl .

      Vous devez traiter les images intermédiaires et écrire l'image résultante sur la surface de sortie.

    • Utiliser le partage de surface Camera2 : utilisez le partage de surface avec une autre surface en ajoutant n'importe quelle instance Camera2OutputConfigImpl à la méthode getSurfaceSharingOutputConfigs() d'une autre instance Camera2OutputConfigImpl . Le format et la taille de la surface doivent être identiques.

    Tous Camera2OutputConfigImpl , y compris SurfaceOutputConfigImpl et ImageReaderOutputConfigImpl , doivent avoir un ID unique ( getId() ), qui est utilisé pour spécifier la surface cible et récupérer l'image à partir de ImageReaderOutputConfigImpl .

  3. onCaptureSessionStart et RequestProcessorImpl

    Lorsque CameraCaptureSession démarre et que le framework Camera appelle onConfigured() , Camera2/X appelle SessionProcessorImpl.onCaptureSessionStart() avec le wrapper de requête Camera2 RequestProcessImpl . Camera2/X implémente RequestProcessImpl , qui vous permet d' exécuter les requêtes de capture et de récupérer des images si ImageReaderOutputConfigImpl est utilisé.

    Les API RequestProcessImpl sont similaires aux API Camera2 CameraCaptureSession en termes d'exécution des requêtes. Les différences sont :

    • La surface cible est spécifiée par l'ID de l'instance Camera2OutputConfigImpl .
    • La capacité de récupérer l'image de ImageReader .

    Vous pouvez appeler RequestProcessorImpl.setImageProcessor() avec un ID Camera2OutputConfigImpl spécifié pour enregistrer une instance ImageProcessorImpl afin de recevoir des images.

    L'instance RequestProcessImpl devient invalide après que Camera2/X appelle SessionProcessorImpl.onCaptureSessionEnd() .

  4. Lancer l'aperçu et prendre une photo

    Dans l'implémentation d'Advanced Extender, vous pouvez envoyer des demandes de capture via l'interface RequestProcessorImpl . Camera2/X vous avertit de démarrer la demande répétée d'aperçu ou la séquence de capture fixe en appelant respectivement SessionProcessorImpl#startRepeating et SessionProcessorImpl#startCapture . Vous devez envoyer des demandes de capture pour satisfaire ces demandes de prévisualisation et de capture fixe.

    Camera2/X définit également les paramètres de demande de capture via SessionProcessorImpl#setParameters . Vous devez définir ces paramètres de demande (si les paramètres sont pris en charge) sur les demandes répétées et uniques.

    Vous devez prendre en charge au moins CaptureRequest.JPEG_ORIENTATION et CaptureRequest.JPEG_QUALITY . extensions-interface 1.3.0 prend en charge les clés de requête et de résultat, qui sont exposées par les méthodes suivantes :

    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys()
    • AdvancedExtenderImpl.getAvailableCaptureResultKeys()

    Lorsque les développeurs définissent les clés dans la liste getAvailableCaptureRequestKeys , vous devez activer les paramètres et vous assurer que le résultat de la capture contient les clés dans la liste getAvailableCaptureResultKeys .

  5. startTrigger

    SessionProcessorImpl.startTrigger() est appelé pour démarrer le déclencheur tel que CaptureRequest.CONTROL_AF_TRIGGER et CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER . Vous pouvez ignorer toutes les clés de demande de capture qui n'ont pas été annoncées dans AdvancedExtenderImpl.getAvailableCaptureRequestKeys() .

    startTrigger() est pris en charge depuis extensions-interface 1.3.0. Il permet aux applications d'implémenter le tap-to-focus et le flash avec des extensions.

  6. Nettoyer

    À la fin d'une session de capture, SessionProcessorImpl.onCaptureSessionEnd() est invoqué avant la fermeture CameraCaptureSession . Une fois la session de capture fermée, deInitSession() effectue le nettoyage.

Prise en charge de l'aperçu, de la capture d'images fixes et de l'analyse d'images

Vous devez appliquer l'extension à la fois pour les cas d'utilisation de l'aperçu et de la capture. Cependant, si la latence est trop élevée pour afficher l'aperçu en douceur, vous pouvez appliquer l'extension uniquement pour la capture fixe.

Pour le type Basic Extender, indépendamment de l'activation de l'extension pour l'aperçu, vous devez implémenter à la fois ImageCaptureExtenderImpl et PreviewExtenderImpl pour une extension donnée. Often, an app also uses a YUV stream to analyze the image content such as finding QR codes or text. To better support this use case , you should support the stream combination of preview, still capture, and a YUV_420_888 stream for configuring CameraCaptureSession . This means that if you implement a processor, then you have to support the stream combination of three YUV_420_888 streams.

For Advanced Extender, Camera2/X passes three output surfaces to the SessionProcessorImpl.initSession() call. These output surfaces are for preview , still capture, and image analysis, respectively. You must ensure that preview and still capture output surfaces show the valid output. However, for the image analysis output surface, ensure it's working only when it's non-null. If your implementation can't support the image analysis stream, you can return an empty list in AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() . This ensures the image analysis output surface is always null in SessionProcessorImpl.initSession() .

Support video capture

The current Camera Extension architecture supports only the preview and still capture use cases. We don't support enabling the extension on the MediaCodec or MediaRecorder surfaces for recording the video. However, it's possible for apps to record the preview output.

Supporting MediaCodec and MediaRecorder surfaces is under investigation.

Extensions interface version history

The following table shows the Camera Extension interface version history. You should always implement the vendor library with the latest version.

Version Added features
1.0.0
  • Version verification
    • ExtensionVersionImpl
  • Basic Extender
    • PreviewExtenderImpl
    • ImageCaptureExtenderImpl
    • Processor
      • PreviewImageProcessorImpl
      • CaptureProcessorImpl
      • RequestUpdateProcessorImpl
1.1.0
  • Library initialization
    • InitializerImpl
  • Expose supported resolutions
    • PreviewExtenderImpl.getSupportedResolutions
    • ImageCaptureExtenderImpl.getSupportedResolutions
1.2.0
  • AdvancedExtender
    • AdvancedExtenderImpl
    • SessionProcessorImpl
  • Get estimated capture latency
    • ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange
1.3.0
  • Expose supported capture request keys/results keys
    • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys
    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys
    • New process() call that takes ProcessResultImpl in PreviewImageProcessorImpl and CaptureProcessorImpl
    • Support trigger type request
      • AdvancedExtenderImpl.startTrigger

Reference implementation

The following reference OEM vendor library implementations are available in frameworks/ex .

  • advancedSample : A basic implementation of Advanced Extender.

  • sample : A basic implementation of Basic Extender.

  • service_based_sample : An implementation that demonstrates how to host Camera Extensions in a Service . This implementation contains the following components:

    • oem_library : A Camera Extensions OEM library for Camera2 and CameraX Extensions APIs that implements Extensions-Interface . This acts as a passthrough that forwards calls from Extensions-Interface to the service. This library also provides AIDL files and wrapper classes to communicate with the service.

      Advanced Extender is enabled by default. To enable the Basic Extender, change ExtensionsVersionImpl#isAdvancedExtenderImplemented to return false .

    • extensions_service : A sample implementation of the Extensions Service. Add your implementation here. The interface to implement in the service is similar to the Extensions-Interface . For example, implementing the IAdvancedExtenderImpl.Stub performs the same operations as AdvancedExtenderImpl . ImageWrapper and TotalCaptureResultWrapper are required to make Image and TotalCaptureResult parcelable.

Set up the vendor library on a device

The OEM vendor library isn't built into an app; it's loaded from the device at runtime by Camera2/X. In CameraX, the <uses-library> tag declares that the androidx.camera.extensions.impl library, which is defined in the AndroidManifest.xml file of the camera-extensions library, is a dependency of CameraX and must be loaded at runtime. In Camera2, the framework loads an extensions service that also declares that the <uses-library> loads the same androidx.camera.extensions.impl library at runtime.

This allows third-party apps using extensions to automatically load the OEM vendor library. The OEM library is marked as optional so apps can run on devices that don't have the library on the device. Camera2/X handles this behavior automatically when an app tries to use a camera extension as long as the device manufacturer places the OEM library on the device so that it can be discovered by the app.

To set up the OEM library on a device, do the following:

  1. Add a permission file, which is required by the <uses-library> tag, using the following format: /etc/permissions/ ANY_FILENAME .xml . For example, /etc/permissions/camera_extensions.xml . The files in this directory provide a mapping of the library named in <uses-library> to the actual file path on the device.

  2. Use the example below to add the required information to the file.

    • name must be androidx.camera.extensions.impl as that's the library that CameraX searches for.
    • file is the absolute path of the file that contains the extensions implementation (for example, /system/framework/androidx.camera.extensions.impl.jar ).
    <?xml version="1.0" encoding="utf-8"?>
<permissions>
    <library name="androidx.camera.extensions.impl"
             file="OEM_IMPLEMENTED_JAR" />
</permissions>

In Android 12 or higher, devices supporting CameraX extensions must have the ro.camerax.extensions.enabled property set to true , which allows for querying whether a device supports extensions. To do this, add the following line in the device make file:

PRODUCT_VENDOR_PROPERTIES += \
    ro.camerax.extensions.enabled=true \

Validation

To test your implementation of the OEM vendor library during the development stage, use the example app at androidx-main/camera/integration-tests/extensionstestapp/ , which runs through various vendor extensions.

After you complete your implementation, use the Camera Extensions Validation Tool to run automated and manual tests to verify that the vendor library is implemented correctly.

Extended scene mode versus Camera Extensions

For the bokeh extension, in addition to exposing it using Camera Extensions, you can expose the extension using the extended scene mode, which is enabled through the CONTROL_EXTENDED_SCENE_MODE key. For more implementation details, see Camera Bokeh .

Extended scene mode has fewer restrictions compared to Camera Extensions for camera2 apps. For example, you can enable extended scene mode in a regular CameraCaptureSession instance that supports flexible stream combinations and capture request parameters. In contrast, camera extensions support only a fixed set of stream types and have limited support for capture request parameters.

A downside of extended scene mode is that you can only implement it in the camera HAL, which means that it must be verified to work across all orthogonal controls available to app developers.

We recommend exposing bokeh using both the extended scene mode and Camera Extensions because apps might prefer to use a particular API to enable bokeh. We recommend first using the extended scene mode because this is the most flexible way for apps to enable the bokeh extension. Then you can implement the camera extensions interface based on the extended scene mode. If implementing bokeh in the camera HAL is difficult, for example, because it requires a post processor running in the app layer to process images, we recommend implementing the bokeh extension using the Camera Extensions interface.

Frequently asked questions (FAQs)

Are there any restrictions on API levels?

Yes. This depends on the Android API feature set that's required by the OEM vendor library implementation. For example, ExtenderStateListener.onPresetSession() uses the SessionConfiguration.setSessionParameters() call to set a baseline set of tags. This call is available only on API level 28 and higher. For details on specific interface methods, see the API reference documentation .

,

Device manufacturers can expose extensions such as bokeh, night mode, and HDR to third-party developers through the Camera Extensions interface provided by the OEM vendor library. Developers can use the Camera2 Extensions API and the CameraX Extensions API to access the extensions implemented in the OEM vendor library.

For a list of supported extensions, which is the same across Camera2 and CameraX, see CameraX Extensions API . If you want to add an extension, file a bug with the Issue Tracker .

This page describes how to implement and enable the OEM vendor library on devices.

Architecture

The following diagram describes the architecture of the Camera Extensions interface or extensions-interface : Architecture

Figure 1. Camera Extensions architecture diagram

As shown in the diagram, to support Camera Extensions, you need to implement the extensions-interface provided by the OEM vendor library. Your OEM vendor library enables two APIs: CameraX Extensions API and Camera2 Extensions API , which are used by CameraX and Camera2 apps, respectively, to access vendor extensions.

Implement the OEM vendor library

To implement the OEM vendor library, copy the camera-extensions-stub files into a system library project. These files define the Camera Extensions interface.

The camera-extensions-stub files are divided into the following categories:

Essential interface files (don't modify)

  • PreviewExtenderImpl.java
  • ImageCaptureExtenderImpl.java
  • ExtenderStateListener.java
  • ProcessorImpl.java
  • PreviewImageProcessorImpl.java
  • CaptureProcessorImpl.java
  • CaptureStageImpl.java
  • RequestUpdateProcessorImpl.java
  • ProcessResultImpl.java
  • advanced/AdvancedExtenderImpl.java
  • advanced/Camera2OutputConfigImpl.java
  • advanced/Camera2SessionConfigImpl.java
  • advanced/ImageProcessorImpl.java
  • advanced/ImageReaderOutputConfigImpl.java
  • advanced/ImageReferenceImpl.java
  • advanced/MultiResolutionImageReaderOutputConfigImpl.java
  • advanced/OutputSurfaceImpl.java
  • advanced/RequestProcessorImpl.java
  • advanced/SessionProcessorImpl.java
  • advanced/SurfaceOutputConfigImpl.java

Mandatory implementations (add your implementation)

  • ExtensionVersionImpl.java
  • InitializerImpl.java

Bokeh extender classes (implement it if Bokeh extension is supported)

  • BokehImageCaptureExtenderImpl.java
  • BokehPreviewExtenderImpl.java
  • advanced/BokehAdvancedExtenderImpl.java

Night extender classes (implement it if Night extension is supported)

  • NightImageCaptureExtenderImpl.java
  • NightPreviewExtenderImpl.java
  • advanced/NightAdvancedExtenderImpl.java

Auto extender classes (implement it if Auto extension is supported)

  • AutoImageCaptureExtenderImpl.java
  • AutoPreviewExtenderImpl.java
  • advanced/AutoAdvancedExtenderImpl.java

HDR extender classes (implement it if HDR extension is supported)

  • HdrImageCaptureExtenderImpl.java
  • HdrPreviewExtenderImpl.java
  • advanced/HdrAdvancedExtenderImpl.java

Face Retouch extender classes (implement it if Face Retouch extension is supported)

  • BeautyImageCaptureExtenderImpl.java
  • BeautyPreviewExtenderImpl.java
  • advanced/BeautyAdvancedExtenderImpl.java

Utilities (optional, can be deleted)

  • advanced/Camera2OutputConfigImplBuilder.java
  • advanced/Camera2SessionConfigImplBuilder.java

You aren't required to provide an implementation for every extension. If you don't implement an extension, set isExtensionAvailable() to return false or remove the corresponding Extender classes. The Camera2 and CameraX Extensions APIs report to the app that the extension is unavailable.

Let's walk through how the Camera2 and CameraX Extensions APIs interact with the vendor library to enable an extension. The following diagram illustrates the end-to-end flow using the Night extension as an example:

Mainflow

Figure 2. Night extension implementation

  1. Version verification:

    Camera2/X calls ExtensionVersionImpl.checkApiVersion() to ensure that the OEM-implemented extensions-interface version is compatible with Camera2/X supported versions.

  2. Vendor library initialization:

    InitializerImpl has a method init() that initializes the vendor library. Camera2/X completes the initialization before accessing the Extender classes.

  3. Instantiate Extender classes:

    Instantiates the Extender classes for the extension. There are two Extender types: Basic Extender and Advanced Extender. You must implement one Extender type for all Extensions. For more information, see Basic Extender versus Advanced Extender .

    Camera2/X instantiates and interacts with the Extender classes to retrieve information and enable the extension. For a given extension, Camera2/X can instantiate the Extender classes multiple times. As a result, don't do heavy-lifting initialization in the constructor or the init() call. Do the heavy lifting only when the camera session is about to start, such as when onInit() is called in Basic Extender or initSession() is called in Advanced Extender.

    For the Night extension, the following Extender classes are instantiated for the Basic Extender type:

    • NightImageCaptureExtenderImpl.java
    • NightPreviewExtenderImpl.java

    And for the Advanced Extender type:

    • NightAdvancedExtenderImpl.java

  4. Check extension availability:

    Before enabling the extension, isExtensionAvailable() checks if the extension is available on the specified camera ID through the Extender instance.

  5. Initialize the Extender with camera information:

    Camera2/X calls init() on the Extender instance and passes it the camera ID and CameraCharacteristics .

  6. Query information:

    Invokes the Extender class to retrieve information such as supported resolutions, still capture estimated latency, and capture request keys from the Extender in preparation for enabling the extension.

  7. Enable extension on the Extender:

    The Extender class provides all the interfaces needed to enable the class. It offers a mechanism to hook OEM implementation into the Camera2 pipeline such as injecting capture request parameters or enabling a post processor.

    For the Advanced Extender type, Camera2/X interacts with SessionProcessorImpl to enable the extension. Camera2/X retrieves the SessionProcessorImpl instance by calling createSessionProcessor() on the Extender.

The following sections describe the extension flow in greater detail.

Version verification

When loading the OEM vendor library from the device at runtime, Camera2/X verifies if the library is compatible with the extensions-interface version. The extensions-interface uses semantic versioning, or MAJOR.MINOR.PATCH, for example, 1.1.0 or 1.2.0. However, only the major and minor versions are used during the version verification.

To verify the version, Camera2/X calls ExtensionVersionImpl.checkApiVersion() with the supported extensions-interface version. Camera2/X then uses the version reported by the OEM library to determine if the extension can be enabled and what capabilities it should invoke.

Major version compatibility

If the major versions of the extension-interface are different between Camera2/X and the vendor library, then it's considered incompatible and the extension is disabled.

Backward compatibility

As long as the major version is identical, Camera2/X ensures backward compatibility with OEM vendor libraries built with prior extensions-interface versions. For example, if Camera2/X supports extensions-interface 1.3.0, the OEM vendor libraries that implemented 1.0.0, 1.1.0, and 1.2.0 are still compatible. This also means that after you implement a specific version of the vendor library, Camera2/X makes sure the library is backward compatible with upcoming extension-interface versions.

Forward compatibility

Forward compatibility with vendor libraries of newer extensions-interface depends on you, the OEM. If you need some features to implement the extensions, you might want to enable the extensions starting from a certain version. In this case, you can return the supported extensions-interface version when the Camera2/X library version meets the requirements. If the Camera2/X versions aren't supported, you can return an incompatible version such as 99.0.0 to disable the extensions.

Vendor library initialization

After verifying the extensions-interface version implemented by the OEM library, Camera2/X starts the initialization process. The InitializerImpl.init() method signals to the OEM library that an app is trying to use extensions.

Camera2/X makes no other calls to the OEM library (aside from version checking) until the OEM vendor library calls OnExtensionsInitializedCallback.onSuccess() to notify the completion of initialization.

You must implement InitializerImpl as of extensions-interface 1.1.0. Camera2/X skips the library initialization step if the OEM vendor library implements extensions-interface 1.0.0.

Basic Extender versus Advanced Extender

There are two types of extensions-interface implementation: Basic Extender and Advanced Extender. Advanced Extender has been supported since extensions-interface 1.2.0.

Implement Basic Extender for extensions that process images in the camera HAL or use a post processor capable of processing YUV streams.

Implement Advanced Extender for extensions that need to customize the Camera2 stream configuration and send capture requests as needed.

See the following table for the comparison:

Basic Extender Advanced Extender
Stream configurations Fixed
Preview: PRIVATE or YUV_420_888 (if processor exists)
Still capture: JPEG or YUV_420_888 (if processor exists)		 Customizable by OEM.
Sending capture request Only Camera2/X can send capture requests. You can set the parameters to these requests. When the processor is provided for image capture, Camera2/X can send multiple capture requests and send all the images and capture results to the processor. A RequestProcessorImpl instance is provided to you to execute the camera2 capture request and get results and Image.

Camera2/X invokes startRepeating and startCapture on SessionProcessorImpl to signal the OEM to start the repeating request for preview and start the still capture sequence respectively.

Hooks in the camera pipeline
  • onPresetSession provides session parameters.
  • onEnableSession sends a single request right after CameraCaptureSession is configured.
  • onDisableSession sends a single request before CameraCaptureSession is closed.
  • initSession initializes and returns a customized camera2 session configuration for creating the capture session.
  • onCaptureSessionStart is invoked right after CameraCaptureSession is configured.
  • onCaptureSessionEnd is invoked before CameraCaptureSession is closed.
Suitable for Extensions implemented in the camera HAL or in a processor that processes YUV images.
  • Has Camera2-based implementations for the extensions.
  • Needs customized stream configuration such as RAW stream.
  • Needs Interactive capture sequence.
Supported API version Camera2 Extensions: Android 13 or higher
CameraX Extensions: camera-extensions 1.1.0 or higher		 Camera2 Extensions: Android 12L or higher
CameraX Extensions: camera-extensions 1.2.0-alpha03 or higher

App flows

The following table shows three types of app flows and their corresponding Camera Extensions API calls. While Camera2/X provide these APIs, you must properly implement the vendor library to support these flows, which we describe in more detail in a later section.

Camera2 extensions CameraX extensions
Query extension availability CameraExtensionCharacteristics . getSupportedExtensions ExtensionsManager. isExtensionAvailable
Query information CameraExtensionCharacteristics. getExtensionSupportedSizes CameraExtensionCharacteristics. getEstimatedCaptureLatencyRangeMillis CameraExtensionCharacteristics. getAvailableCaptureRequestKeys CameraExtensionCharacteristics. getAvailableCaptureResultKeys ExtensionsManager. getEstimatedCaptureLatencyRange

CameraX handles the rest of the information within the library.

Preview and still-capture with extension enabled CameraDevice. createExtensionSession

cameraExtensionsSession. setRepeatingRequest

cameraExtensionsSession. capture

 val cameraSelector = ExtensionsManager. getExtensionEnabledCameraSelector

bindToLifecycle(lifecycleOwner, cameraSelector, preview, ...)

Basic Extender

The Basic Extender interface provides hooks into several places in the camera pipeline. Each extension type has corresponding Extender classes that OEMs need to implement.

The following table lists the Extender classes OEMS need to implement for each extension:

Extender classes to implement
Night NightPreviewExtenderImpl.java

NightImageCaptureExtenderImpl.java

HDR HdrPreviewExtenderImpl.java

HdrImageCaptureExtenderImpl.java

Auto AutoPreviewExtenderImpl.java

AutoImageCaptureExtenderImpl.java

Bokeh BokehPreviewExtenderImpl.java

BokehImageCaptureExtenderImpl.java

Face retouch BeautyPreviewExtenderImpl.java

BeautyImageCaptureExtenderImpl.java

We use PreviewExtenderImpl and ImageCaptureExtenderImpl as placeholders in the following example. Replace these with the names of the actual files you're implementing.

Basic Extender has the following capabilities:

  • Inject session parameters when configuring CameraCaptureSession ( onPresetSession ).
  • Notify you of the capture session start and closing events and send a single request to notify the HAL with the returned parameters ( onEnableSession , onDisableSession ).
  • Inject capture parameters for the request ( PreviewExtenderImpl.getCaptureStage , ImageCaptureExtenderImpl.getCaptureStages ).
  • Add processors for preview and still capture that's capable of processing YUV_420_888 stream.

Let's see how Camera2/X invokes the extensions-interface to achieve the three app flows mentioned above.

App flow 1: Check extension availability

BasicExtenderAppFlow1

Figure 3. App flow 1 on Basic Extender

In this flow, Camera2/X directly calls the isExtensionAvailable() method of both PreviewExtenderImpl and ImageCaptureExtenderImpl without calling init() . Both Extender classes must return true to enable the extensions.

This is often the first step for apps to check if the given extension type is supported for a given camera ID before enabling the extension. This is because some extensions are supported only on certain camera IDs.

App flow 2: Query information

BasicExtenderAppFlow2

Figure 4. App flow 2 on Basic Extender

After determining if the extension is available, apps should query the following information before enabling the extension.

  • Still capture latency range: ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange returns the range of the capture latency for the app to evaluate if it's appropriate to enable the extension for the current scenario.

  • Supported sizes for the preview and capture surface: ImageCaptureExtenderImpl.getSupportedResolutions and PreviewExtenderImpl.getSupportedResolutions return a list of image formats and the sizes that are supported for surface format and size.

  • Supported request and result keys: Camera2/X invokes the following methods to retrieve the supported capture request keys and result keys from your implementation:

    • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys
    • ImageCaptureExtenderImpl.getAvailableCapturetResultKeys

Camera2/X always calls init() first on these Extender classes before querying for more information.

App flow 3: Preview/still capture with extension enabled (HAL implementation)

BasicExtenderAppFlow3

Figure 5. App flow 3 on Basic Extender

The above diagram illustrates the main flow of enabling preview and still capture with an extension without any processor. This means the camera HAL processes the extension.

In this flow, Camera2/X first calls init() then onInit , which notifies you that a camera session is about to start with the specified extensions. You can do heavy-lifting initialization in onInit() .

When configuring CameraCaptureSession , Camera2/X invokes onPresetSession to get the session parameters. After the capture session is configured successfully, Camera2/X invokes onEnableSession returning a CaptureStageImpl instance that contains the capture parameters. Camera2/X immediately sends a single request with these capture parameters to notify the HAL. Similarly, before the capture session is closed, Camera2/X invokes onDisableSession and then sends a single request with the returned capture parameters.

The repeating request triggered by Camera2/X contains the request parameters returned by PreviewExtenderImpl.getCaptureStage() . Furthermore, the still capture request contains the parameters returned by ImageCaptureExtenderImpl.getCaptureStages() .

Finally, Camera2/X invokes onDeInit() after the camera session has finished. You can release resources in onDeinit() .

Preview processor

In addition to the camera HAL, you can also implement extensions in a processor.

Implement PreviewExtenderImpl.getProcessorType to specify the processor type as explained below:

  • PROCESSOR_TYPE_NONE : No processor. Images are processed in the camera HAL.

  • PROCESSOR_TYPE_REQUEST_UPDATE_ONLY : The processor type lets you update the repeating request with new capture request parameters based on the latest TotalCaptureResult .

    PreviewExtenderImpl.getProcessor must return a RequestUpdateProcessorImpl instance that processes the TotalCaptureResult instance and returns a CaptureStageImpl instance to update the repeating request. PreviewExtenderImpl.getCaptureStage() should also reflect the result of the processing and return the latest CaptureStageImpl .

  • PROCESSOR_TYPE_IMAGE_PROCESSOR : This type allows you to implement a processor to process YUV_420_888 images and write the output to a PRIVATE surface.

    You need to implement and return a PreviewImageProcessorImpl instance in PreviewExtenderImpl.getProcessor . The processor is responsible for processing YUV_420_888 input images. It should write the output to the PRIVATE format of preview. Camera2/X uses a YUV_420_888 surface instead of PRIVATE to configure the CameraCaptureSession for preview.

    See following illustration for the flow:

PreviewProcessor

Figure 6. Preview flow with PreviewImageProcessorImpl

The PreviewImageProcessorImpl interface extends ProcessImpl and has three important methods:

  • onOutputSurface(Surface surface, int imageFormat) sets the output surface for the processor. For PreviewImageProcessorImpl , imageFormat is a pixel format such as PixelFormat.RGBA_8888 .

  • onResolutionUpdate(Size size) sets the size of the input image.

  • onImageFormatUpdate(int imageFormat) sets the image format of the input image. Currently, it can only be YUV_420_888 .

Image capture processor

For still capture, you can implement a processor by returning a CaptureProcessorImpl instance using ImageCaptureExtenderImpl.getCaptureProcessor . The processor is responsible to process a list of captured YUV_420_888 images and TotalCaptureResult instances and write the output to a YUV_420_888 surface.

You can safely assume that preview is enabled and running before sending the still capture request.

See the flow in the diagram below:

CaptureProcessor

Figure 7. Still capture flow with CaptureProcessorImpl

  1. Camera2/X uses a YUV_420_888 format surface for still capture to configure the capture session. Camera2/X prepares CaptureProcessorImpl by calling:

    • CaptureProcessorImpl.onImageFormatUpdate() with YUV_420_888 .
    • CaptureProcessorImpl.onResolutionUpdate() with the input image size.
    • CaptureProcessorImpl.onOutputSurface() with an output YUV_420_888 surface.

  2. ImageCaptureExtenderImpl.getCaptureStages returns a list of CaptureStageImpl , where each element maps to a CaptureRequest instance with capture parameters that are sent by Camera2/X. For example, if it returns a list of three CaptureStageImpl instances, Camera2/X sends three capture requests with corresponding capture parameters using the captureBurst API.

  3. The received images and TotalCaptureResult instances are bundled together and sent to CaptureProcessorImpl for processing.

  4. CaptureProcessorImpl writes the result Image ( YUV_420_888 format) to the output surface specified by the onOutputSurface() call. Camera2/X converts it into JPEG images if necessary.

Support capture request keys and results

In addition to camera preview and capture, apps can set zoom, flash parameters, or trigger a tap-to-focus. These parameters might not be compatible with your extension implementation.

The following methods have been added to extensions-interface 1.3.0 to allow you to expose the parameters that your implementation supports:

  • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys() returns the capture request keys supported by your implementation.
  • ImageCaptureExtenderImpl.getAvailableCaptureResultKeys() returns the capture result keys that are contained in the capture result.

If the camera HAL processes the extension, Camera2/X retrieves the capture results in CameraCaptureSession.CaptureCallback . However, if the processor is implemented, then Camera2/X retrieves the capture results in ProcessResultImpl , which is passed to the process() method in PreviewImageProcessorImpl and CaptureProcessorImpl . You're responsible for reporting the capture result through ProcessResultImpl to Camera2/X.

See the definition of the CaptureProcessorImpl interface below as an example. In extensions-interface 1.3.0 or higher, the second process() call is invoked:

Interface CaptureProcessorImpl extends ProcessorImpl {
    // invoked when extensions-interface version < 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results);
    // invoked when extensions-interface version >= 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results,
            ProcessResultImpl resultCallback, Executor executor);
}

For common camera operations like zoom, tap-to-focus, flash, and exposure compensation, we recommend supporting the following keys for both capture request and capture result:

  • Zoom:
    • CaptureRequest#CONTROL_ZOOM_RATIO
    • CaptureRequest#SCALER_CROP_REGION
  • Tap-to-focus:
    • CaptureRequest#CONTROL_AF_MODE
    • CaptureRequest#CONTROL_AF_TRIGGER
    • CaptureRequest#CONTROL_AF_REGIONS
    • CaptureRequest#CONTROL_AE_REGIONS
    • CaptureRequest#CONTROL_AWB_REGIONS
  • Flash:
    • CaptureRequest#CONTROL_AE_MODE
    • CaptureRequest#CONTROL_AE_PRECAPTURE_TRIGGER
    • CaptureRequest#FLASH_MODE
  • Exposure compensation:
    • CaptureRequest#CONTROL_AE_EXPOSURE_COMPENSATION

For Basic Extenders that implement 1.2.0 or prior versions, the CameraX Extensions API explicitly supports all the above keys. For extensions-interface 1.3.0, both CameraX and Camera2 honor the returned list and support only the keys contained in it. For example, if you decide to return only CaptureRequest#CONTROL_ZOOM_RATIO and CaptureRequest#SCALER_CROP_REGION in the 1.3.0 implementation, then that means only zoom is supported for the app while tap-to-focus, flash, and exposure compensation aren't allowed.

Advanced Extender

Advanced Extender is a type of vendor implementation based on the Camera2 API. This Extender type was added in extensions-interface 1.2.0. Depending on the device manufacturer, extensions might be implemented in the app layer, which depends on the following factors:

  • Custom stream configuration: Configure custom streams like RAW stream or have multiple streams for different physical camera IDs.

  • Capability to send Camera2 requests: Support a complicated interaction logic that can send capture requests with parameters based on the results of previous requests.

Advanced Extender provides a wrapper, or an intermediate layer, so you can customize the stream configuration and send capture requests on demand.

Files to implement

To switch to the Advanced Extender implementation, the isAdvancedExtenderImplemented() method in ExtensionVersionImpl must return true . For each extension type, OEMs must implement the corresponding Extender classes. The Advanced Extender implementation files are in the advanced package.

Extender classes to implement
Night advanced/NightAdvancedExtenderImpl.java
HDR advanced/HdrAdvancedExtenderImpl.java
Auto advanced/AutoAdvancedExtenderImpl.java
Bokeh advanced/BokehAdvancedExtenderImpl.java
Face Retouch advanced/BeautyAdvancedExtenderImpl.java

We use AdvancedExtenderImpl as a placeholder in the following example. Replace it with the name of the Extender file for the extension you're implementing.

Let's see how Camera2/X invokes the extensions-interface to achieve the three app flows.

App flow 1: Check extensions availability

AdvancedAppFlow1

Figure 8. App flow 1 on Advanced Extender

First, the app checks if the given extension is supported.

App flow 2: Query information

AdvancedAppFlow2

Figure 9. App flow 2 on Advanced Extender

After calling AdvancedExtenderImpl.init() , the app can query the following the information on AdvancedExtenderImpl :

  • Estimated still capture latency: AdvancedExtenderImpl.getEstimatedCaptureLatencyRange() returns the range of the capture latency for the app to evaluate if it is appropriate to enable the extension for the current scenario.

  • Supported resolutions for preview and still capture:

    • AdvancedExtenderImpl.getSupportedPreviewOutputResolutions() returns a map of image format to the sizes list that are supported for preview surface format and size. OEMs must support at least the PRIVATE format.

    • AdvancedExtenderImpl.getSupportedCaptureOutputResolutions() returns the supported format and sizes for still capture surface. OEMs must support both JPEG and YUV_420_888 format output.

    • AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() returns the supported sizes for an extra YUV_420_888 stream for image analysis. If the image analysis YUV surface isn't supported, getSupportedYuvAnalysisResolutions() should return null or an empty list.

  • Available capture request keys/results (added in extensions-interface 1.3.0): Camera2/X invokes the following methods to retrieve the supported capture request keys and result keys from your implementation:

    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys
    • AdvancedExtenderImpl.getAvailableCaptureResultKeys

For more information, see Support capture request keys and results .

App flow 3: Preview/still capture with extension enabled

AdvancedAppFlow3

Figure 10. App flow 3 on Advanced Extender

The above diagram shows the main flow for starting preview and still capture for the Advanced Extender type. Let's walk through each step.

  1. SessionProcessorImpl instance

    The core Advanced Extender implementation is in SessionProcessorImpl , which is responsible for providing customized session configuration and sending capture requests to initiate the preview and still capture request. AdvancedExtenderImpl.createSessionProcessor() is invoked to return the SessionProcessorImpl instance.

  2. initSession

    SessionProcessorImpl.initSession() initializes the session for the extension. This is where you allocate resources and return a session configuration for preparing a CameraCaptureSession .

    For the input parameters, Camera2/X specifies the output surface configurations for preview, still capture, and an optional YUV image analysis. This output surface configuration ( OutputSurfaceImpl ) contains the surface, size and image format that are retrieved by following methods in AdvancedExtenderImpl :

    • getSupportedPreviewOutputResolutions()
    • getSupportedCaptureOutputResolutions()
    • getSupportedYuvAnalysisResolutions()

    You must return a Camera2SessionConfigImpl instance, which consists of a list of Camera2OutputConfigImpl instances and the session parameters used for configuring CameraCaptureSession . You're responsible for outputting the correct camera images to the output surfaces passed in by Camera2/X. Here are some options to enable the output:

    • Processing in camera HAL: You can directly add the output surfaces to CameraCaptureSession with a SurfaceOutputConfigImpl implementation. This configures the supplied output surface to the camera pipeline and allows the camera HAL to process the image.

    • Processing intermediate ImageReader surface (RAW, YUV, etc): Add the intermediate ImageReader surfaces to the CameraCaptureSession with an ImageReaderOutputConfigImpl instance.

      You need to process the intermediate images and write the result image to the output surface.

    • Use Camera2 surface sharing: Use surface sharing with another surface by adding any Camera2OutputConfigImpl instance to the getSurfaceSharingOutputConfigs() method of another Camera2OutputConfigImpl instance. The surface format and size must be identical.

    All Camera2OutputConfigImpl including SurfaceOutputConfigImpl and ImageReaderOutputConfigImpl must have a unique ID ( getId() ), which is used to specify the target surface and retrieve the image from ImageReaderOutputConfigImpl .

  3. onCaptureSessionStart and RequestProcessorImpl

    When CameraCaptureSession starts and the Camera framework invokes onConfigured() , then Camera2/X invokes SessionProcessorImpl.onCaptureSessionStart() with the Camera2 request wrapper RequestProcessImpl . Camera2/X implements RequestProcessImpl , which enables you to execute the capture requests , and retrieve images if ImageReaderOutputConfigImpl is used.

    The RequestProcessImpl APIs are similar to the Camera2 CameraCaptureSession APIs in terms of executing requests. The differences are:

    • The target surface is specified by the ID of the Camera2OutputConfigImpl instance.
    • The capability of retrieving the image of the ImageReader .

    You can call RequestProcessorImpl.setImageProcessor() with a specified Camera2OutputConfigImpl ID to register an ImageProcessorImpl instance to receive images.

    The RequestProcessImpl instance becomes invalid after Camera2/X calls SessionProcessorImpl.onCaptureSessionEnd() .

  4. Start the preview and take a picture

    In the Advanced Extender implementation, you can send capture requests through the RequestProcessorImpl interface. Camera2/X notifies you to start the repeating request for preview or the still capture sequence by calling SessionProcessorImpl#startRepeating and SessionProcessorImpl#startCapture respectively. You should send capture requests to satisfy these preview and still-capture requests.

    Camera2/X also sets the capture request parameters through SessionProcessorImpl#setParameters . You must set these request parameters (if parameters are supported) on both the repeating and single requests.

    You must support at least CaptureRequest.JPEG_ORIENTATION and CaptureRequest.JPEG_QUALITY . extensions-interface 1.3.0 supports request and result keys, which are exposed by the following methods:

    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys()
    • AdvancedExtenderImpl.getAvailableCaptureResultKeys()

    When developers set the keys in the getAvailableCaptureRequestKeys list, you must enable the parameters and ensure the capture result contains the keys in the getAvailableCaptureResultKeys list.

  5. startTrigger

    SessionProcessorImpl.startTrigger() is invoked to start the trigger such as CaptureRequest.CONTROL_AF_TRIGGER and CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER . You can disregard any capture request keys that weren't advertised in AdvancedExtenderImpl.getAvailableCaptureRequestKeys() .

    startTrigger() has been supported since extensions-interface 1.3.0. It enables apps to implement tap-to-focus and flash with extensions.

  6. Clean up

    When finishing a capture session, SessionProcessorImpl.onCaptureSessionEnd() is invoked ahead of closing CameraCaptureSession . After the capture session has closed, deInitSession() performs the clean up.

Support preview, still capture, and image analysis

You should apply the extension for both the preview and still capture use cases. However, if the latency is too high to smoothly show the preview, you can apply the extension only for still capture.

For the Basic Extender type, regardless of enabling the extension for preview, you must implement both ImageCaptureExtenderImpl and PreviewExtenderImpl for a given extension. Often, an app also uses a YUV stream to analyze the image content such as finding QR codes or text. To better support this use case , you should support the stream combination of preview, still capture, and a YUV_420_888 stream for configuring CameraCaptureSession . This means that if you implement a processor, then you have to support the stream combination of three YUV_420_888 streams.

For Advanced Extender, Camera2/X passes three output surfaces to the SessionProcessorImpl.initSession() call. These output surfaces are for preview , still capture, and image analysis, respectively. You must ensure that preview and still capture output surfaces show the valid output. However, for the image analysis output surface, ensure it's working only when it's non-null. If your implementation can't support the image analysis stream, you can return an empty list in AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() . This ensures the image analysis output surface is always null in SessionProcessorImpl.initSession() .

Support video capture

The current Camera Extension architecture supports only the preview and still capture use cases. We don't support enabling the extension on the MediaCodec or MediaRecorder surfaces for recording the video. However, it's possible for apps to record the preview output.

Supporting MediaCodec and MediaRecorder surfaces is under investigation.

Extensions interface version history

The following table shows the Camera Extension interface version history. You should always implement the vendor library with the latest version.

Version Added features
1.0.0
  • Version verification
    • ExtensionVersionImpl
  • Basic Extender
    • PreviewExtenderImpl
    • ImageCaptureExtenderImpl
    • Processor
      • PreviewImageProcessorImpl
      • CaptureProcessorImpl
      • RequestUpdateProcessorImpl
1.1.0
  • Library initialization
    • InitializerImpl
  • Expose supported resolutions
    • PreviewExtenderImpl.getSupportedResolutions
    • ImageCaptureExtenderImpl.getSupportedResolutions
1.2.0
  • AdvancedExtender
    • AdvancedExtenderImpl
    • SessionProcessorImpl
  • Get estimated capture latency
    • ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange
1.3.0
  • Expose supported capture request keys/results keys
    • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys
    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys
    • New process() call that takes ProcessResultImpl in PreviewImageProcessorImpl and CaptureProcessorImpl
    • Support trigger type request
      • AdvancedExtenderImpl.startTrigger

Reference implementation

The following reference OEM vendor library implementations are available in frameworks/ex .

  • advancedSample : A basic implementation of Advanced Extender.

  • sample : A basic implementation of Basic Extender.

  • service_based_sample : An implementation that demonstrates how to host Camera Extensions in a Service . This implementation contains the following components:

    • oem_library : A Camera Extensions OEM library for Camera2 and CameraX Extensions APIs that implements Extensions-Interface . This acts as a passthrough that forwards calls from Extensions-Interface to the service. This library also provides AIDL files and wrapper classes to communicate with the service.

      Advanced Extender is enabled by default. To enable the Basic Extender, change ExtensionsVersionImpl#isAdvancedExtenderImplemented to return false .

    • extensions_service : A sample implementation of the Extensions Service. Add your implementation here. The interface to implement in the service is similar to the Extensions-Interface . For example, implementing the IAdvancedExtenderImpl.Stub performs the same operations as AdvancedExtenderImpl . ImageWrapper and TotalCaptureResultWrapper are required to make Image and TotalCaptureResult parcelable.

Set up the vendor library on a device

The OEM vendor library isn't built into an app; it's loaded from the device at runtime by Camera2/X. In CameraX, the <uses-library> tag declares that the androidx.camera.extensions.impl library, which is defined in the AndroidManifest.xml file of the camera-extensions library, is a dependency of CameraX and must be loaded at runtime. In Camera2, the framework loads an extensions service that also declares that the <uses-library> loads the same androidx.camera.extensions.impl library at runtime.

This allows third-party apps using extensions to automatically load the OEM vendor library. The OEM library is marked as optional so apps can run on devices that don't have the library on the device. Camera2/X handles this behavior automatically when an app tries to use a camera extension as long as the device manufacturer places the OEM library on the device so that it can be discovered by the app.

To set up the OEM library on a device, do the following:

  1. Add a permission file, which is required by the <uses-library> tag, using the following format: /etc/permissions/ ANY_FILENAME .xml . For example, /etc/permissions/camera_extensions.xml . The files in this directory provide a mapping of the library named in <uses-library> to the actual file path on the device.

  2. Use the example below to add the required information to the file.

    • name must be androidx.camera.extensions.impl as that's the library that CameraX searches for.
    • file is the absolute path of the file that contains the extensions implementation (for example, /system/framework/androidx.camera.extensions.impl.jar ).
    <?xml version="1.0" encoding="utf-8"?>
<permissions>
    <library name="androidx.camera.extensions.impl"
             file="OEM_IMPLEMENTED_JAR" />
</permissions>

In Android 12 or higher, devices supporting CameraX extensions must have the ro.camerax.extensions.enabled property set to true , which allows for querying whether a device supports extensions. To do this, add the following line in the device make file:

PRODUCT_VENDOR_PROPERTIES += \
    ro.camerax.extensions.enabled=true \

Validation

To test your implementation of the OEM vendor library during the development stage, use the example app at androidx-main/camera/integration-tests/extensionstestapp/ , which runs through various vendor extensions.

After you complete your implementation, use the Camera Extensions Validation Tool to run automated and manual tests to verify that the vendor library is implemented correctly.

Extended scene mode versus Camera Extensions

For the bokeh extension, in addition to exposing it using Camera Extensions, you can expose the extension using the extended scene mode, which is enabled through the CONTROL_EXTENDED_SCENE_MODE key. For more implementation details, see Camera Bokeh .

Extended scene mode has fewer restrictions compared to Camera Extensions for camera2 apps. For example, you can enable extended scene mode in a regular CameraCaptureSession instance that supports flexible stream combinations and capture request parameters. In contrast, camera extensions support only a fixed set of stream types and have limited support for capture request parameters.

A downside of extended scene mode is that you can only implement it in the camera HAL, which means that it must be verified to work across all orthogonal controls available to app developers.

We recommend exposing bokeh using both the extended scene mode and Camera Extensions because apps might prefer to use a particular API to enable bokeh. We recommend first using the extended scene mode because this is the most flexible way for apps to enable the bokeh extension. Then you can implement the camera extensions interface based on the extended scene mode. If implementing bokeh in the camera HAL is difficult, for example, because it requires a post processor running in the app layer to process images, we recommend implementing the bokeh extension using the Camera Extensions interface.

Frequently asked questions (FAQs)

Are there any restrictions on API levels?

Yes. This depends on the Android API feature set that's required by the OEM vendor library implementation. For example, ExtenderStateListener.onPresetSession() uses the SessionConfiguration.setSessionParameters() call to set a baseline set of tags. This call is available only on API level 28 and higher. For details on specific interface methods, see the API reference documentation .

,

Device manufacturers can expose extensions such as bokeh, night mode, and HDR to third-party developers through the Camera Extensions interface provided by the OEM vendor library. Developers can use the Camera2 Extensions API and the CameraX Extensions API to access the extensions implemented in the OEM vendor library.

For a list of supported extensions, which is the same across Camera2 and CameraX, see CameraX Extensions API . If you want to add an extension, file a bug with the Issue Tracker .

This page describes how to implement and enable the OEM vendor library on devices.

Architecture

The following diagram describes the architecture of the Camera Extensions interface or extensions-interface : Architecture

Figure 1. Camera Extensions architecture diagram

As shown in the diagram, to support Camera Extensions, you need to implement the extensions-interface provided by the OEM vendor library. Your OEM vendor library enables two APIs: CameraX Extensions API and Camera2 Extensions API , which are used by CameraX and Camera2 apps, respectively, to access vendor extensions.

Implement the OEM vendor library

To implement the OEM vendor library, copy the camera-extensions-stub files into a system library project. These files define the Camera Extensions interface.

The camera-extensions-stub files are divided into the following categories:

Essential interface files (don't modify)

  • PreviewExtenderImpl.java
  • ImageCaptureExtenderImpl.java
  • ExtenderStateListener.java
  • ProcessorImpl.java
  • PreviewImageProcessorImpl.java
  • CaptureProcessorImpl.java
  • CaptureStageImpl.java
  • RequestUpdateProcessorImpl.java
  • ProcessResultImpl.java
  • advanced/AdvancedExtenderImpl.java
  • advanced/Camera2OutputConfigImpl.java
  • advanced/Camera2SessionConfigImpl.java
  • advanced/ImageProcessorImpl.java
  • advanced/ImageReaderOutputConfigImpl.java
  • advanced/ImageReferenceImpl.java
  • advanced/MultiResolutionImageReaderOutputConfigImpl.java
  • advanced/OutputSurfaceImpl.java
  • advanced/RequestProcessorImpl.java
  • advanced/SessionProcessorImpl.java
  • advanced/SurfaceOutputConfigImpl.java

Mandatory implementations (add your implementation)

  • ExtensionVersionImpl.java
  • InitializerImpl.java

Bokeh extender classes (implement it if Bokeh extension is supported)

  • BokehImageCaptureExtenderImpl.java
  • BokehPreviewExtenderImpl.java
  • advanced/BokehAdvancedExtenderImpl.java

Night extender classes (implement it if Night extension is supported)

  • NightImageCaptureExtenderImpl.java
  • NightPreviewExtenderImpl.java
  • advanced/NightAdvancedExtenderImpl.java

Auto extender classes (implement it if Auto extension is supported)

  • AutoImageCaptureExtenderImpl.java
  • AutoPreviewExtenderImpl.java
  • advanced/AutoAdvancedExtenderImpl.java

HDR extender classes (implement it if HDR extension is supported)

  • HdrImageCaptureExtenderImpl.java
  • HdrPreviewExtenderImpl.java
  • advanced/HdrAdvancedExtenderImpl.java

Face Retouch extender classes (implement it if Face Retouch extension is supported)

  • BeautyImageCaptureExtenderImpl.java
  • BeautyPreviewExtenderImpl.java
  • advanced/BeautyAdvancedExtenderImpl.java

Utilities (optional, can be deleted)

  • advanced/Camera2OutputConfigImplBuilder.java
  • advanced/Camera2SessionConfigImplBuilder.java

You aren't required to provide an implementation for every extension. If you don't implement an extension, set isExtensionAvailable() to return false or remove the corresponding Extender classes. The Camera2 and CameraX Extensions APIs report to the app that the extension is unavailable.

Let's walk through how the Camera2 and CameraX Extensions APIs interact with the vendor library to enable an extension. The following diagram illustrates the end-to-end flow using the Night extension as an example:

Mainflow

Figure 2. Night extension implementation

  1. Version verification:

    Camera2/X calls ExtensionVersionImpl.checkApiVersion() to ensure that the OEM-implemented extensions-interface version is compatible with Camera2/X supported versions.

  2. Vendor library initialization:

    InitializerImpl has a method init() that initializes the vendor library. Camera2/X completes the initialization before accessing the Extender classes.

  3. Instantiate Extender classes:

    Instantiates the Extender classes for the extension. There are two Extender types: Basic Extender and Advanced Extender. You must implement one Extender type for all Extensions. For more information, see Basic Extender versus Advanced Extender .

    Camera2/X instantiates and interacts with the Extender classes to retrieve information and enable the extension. For a given extension, Camera2/X can instantiate the Extender classes multiple times. As a result, don't do heavy-lifting initialization in the constructor or the init() call. Do the heavy lifting only when the camera session is about to start, such as when onInit() is called in Basic Extender or initSession() is called in Advanced Extender.

    For the Night extension, the following Extender classes are instantiated for the Basic Extender type:

    • NightImageCaptureExtenderImpl.java
    • NightPreviewExtenderImpl.java

    And for the Advanced Extender type:

    • NightAdvancedExtenderImpl.java

  4. Check extension availability:

    Before enabling the extension, isExtensionAvailable() checks if the extension is available on the specified camera ID through the Extender instance.

  5. Initialize the Extender with camera information:

    Camera2/X calls init() on the Extender instance and passes it the camera ID and CameraCharacteristics .

  6. Query information:

    Invokes the Extender class to retrieve information such as supported resolutions, still capture estimated latency, and capture request keys from the Extender in preparation for enabling the extension.

  7. Enable extension on the Extender:

    The Extender class provides all the interfaces needed to enable the class. It offers a mechanism to hook OEM implementation into the Camera2 pipeline such as injecting capture request parameters or enabling a post processor.

    For the Advanced Extender type, Camera2/X interacts with SessionProcessorImpl to enable the extension. Camera2/X retrieves the SessionProcessorImpl instance by calling createSessionProcessor() on the Extender.

The following sections describe the extension flow in greater detail.

Version verification

When loading the OEM vendor library from the device at runtime, Camera2/X verifies if the library is compatible with the extensions-interface version. The extensions-interface uses semantic versioning, or MAJOR.MINOR.PATCH, for example, 1.1.0 or 1.2.0. However, only the major and minor versions are used during the version verification.

To verify the version, Camera2/X calls ExtensionVersionImpl.checkApiVersion() with the supported extensions-interface version. Camera2/X then uses the version reported by the OEM library to determine if the extension can be enabled and what capabilities it should invoke.

Major version compatibility

If the major versions of the extension-interface are different between Camera2/X and the vendor library, then it's considered incompatible and the extension is disabled.

Backward compatibility

As long as the major version is identical, Camera2/X ensures backward compatibility with OEM vendor libraries built with prior extensions-interface versions. For example, if Camera2/X supports extensions-interface 1.3.0, the OEM vendor libraries that implemented 1.0.0, 1.1.0, and 1.2.0 are still compatible. This also means that after you implement a specific version of the vendor library, Camera2/X makes sure the library is backward compatible with upcoming extension-interface versions.

Forward compatibility

Forward compatibility with vendor libraries of newer extensions-interface depends on you, the OEM. If you need some features to implement the extensions, you might want to enable the extensions starting from a certain version. In this case, you can return the supported extensions-interface version when the Camera2/X library version meets the requirements. If the Camera2/X versions aren't supported, you can return an incompatible version such as 99.0.0 to disable the extensions.

Vendor library initialization

After verifying the extensions-interface version implemented by the OEM library, Camera2/X starts the initialization process. The InitializerImpl.init() method signals to the OEM library that an app is trying to use extensions.

Camera2/X makes no other calls to the OEM library (aside from version checking) until the OEM vendor library calls OnExtensionsInitializedCallback.onSuccess() to notify the completion of initialization.

You must implement InitializerImpl as of extensions-interface 1.1.0. Camera2/X skips the library initialization step if the OEM vendor library implements extensions-interface 1.0.0.

Basic Extender versus Advanced Extender

There are two types of extensions-interface implementation: Basic Extender and Advanced Extender. Advanced Extender has been supported since extensions-interface 1.2.0.

Implement Basic Extender for extensions that process images in the camera HAL or use a post processor capable of processing YUV streams.

Implement Advanced Extender for extensions that need to customize the Camera2 stream configuration and send capture requests as needed.

See the following table for the comparison:

Basic Extender Advanced Extender
Stream configurations Fixed
Preview: PRIVATE or YUV_420_888 (if processor exists)
Still capture: JPEG or YUV_420_888 (if processor exists)		 Customizable by OEM.
Sending capture request Only Camera2/X can send capture requests. You can set the parameters to these requests. When the processor is provided for image capture, Camera2/X can send multiple capture requests and send all the images and capture results to the processor. A RequestProcessorImpl instance is provided to you to execute the camera2 capture request and get results and Image.

Camera2/X invokes startRepeating and startCapture on SessionProcessorImpl to signal the OEM to start the repeating request for preview and start the still capture sequence respectively.

Hooks in the camera pipeline
  • onPresetSession provides session parameters.
  • onEnableSession sends a single request right after CameraCaptureSession is configured.
  • onDisableSession sends a single request before CameraCaptureSession is closed.
  • initSession initializes and returns a customized camera2 session configuration for creating the capture session.
  • onCaptureSessionStart is invoked right after CameraCaptureSession is configured.
  • onCaptureSessionEnd is invoked before CameraCaptureSession is closed.
Suitable for Extensions implemented in the camera HAL or in a processor that processes YUV images.
  • Has Camera2-based implementations for the extensions.
  • Needs customized stream configuration such as RAW stream.
  • Needs Interactive capture sequence.
Supported API version Camera2 Extensions: Android 13 or higher
CameraX Extensions: camera-extensions 1.1.0 or higher		 Camera2 Extensions: Android 12L or higher
CameraX Extensions: camera-extensions 1.2.0-alpha03 or higher

App flows

The following table shows three types of app flows and their corresponding Camera Extensions API calls. While Camera2/X provide these APIs, you must properly implement the vendor library to support these flows, which we describe in more detail in a later section.

Camera2 extensions CameraX extensions
Query extension availability CameraExtensionCharacteristics . getSupportedExtensions ExtensionsManager. isExtensionAvailable
Query information CameraExtensionCharacteristics. getExtensionSupportedSizes CameraExtensionCharacteristics. getEstimatedCaptureLatencyRangeMillis CameraExtensionCharacteristics. getAvailableCaptureRequestKeys CameraExtensionCharacteristics. getAvailableCaptureResultKeys ExtensionsManager. getEstimatedCaptureLatencyRange

CameraX handles the rest of the information within the library.

Preview and still-capture with extension enabled CameraDevice. createExtensionSession

cameraExtensionsSession. setRepeatingRequest

cameraExtensionsSession. capture

 val cameraSelector = ExtensionsManager. getExtensionEnabledCameraSelector

bindToLifecycle(lifecycleOwner, cameraSelector, preview, ...)

Basic Extender

The Basic Extender interface provides hooks into several places in the camera pipeline. Each extension type has corresponding Extender classes that OEMs need to implement.

The following table lists the Extender classes OEMS need to implement for each extension:

Extender classes to implement
Night NightPreviewExtenderImpl.java

NightImageCaptureExtenderImpl.java

HDR HdrPreviewExtenderImpl.java

HdrImageCaptureExtenderImpl.java

Auto AutoPreviewExtenderImpl.java

AutoImageCaptureExtenderImpl.java

Bokeh BokehPreviewExtenderImpl.java

BokehImageCaptureExtenderImpl.java

Face retouch BeautyPreviewExtenderImpl.java

BeautyImageCaptureExtenderImpl.java

We use PreviewExtenderImpl and ImageCaptureExtenderImpl as placeholders in the following example. Replace these with the names of the actual files you're implementing.

Basic Extender has the following capabilities:

  • Inject session parameters when configuring CameraCaptureSession ( onPresetSession ).
  • Notify you of the capture session start and closing events and send a single request to notify the HAL with the returned parameters ( onEnableSession , onDisableSession ).
  • Inject capture parameters for the request ( PreviewExtenderImpl.getCaptureStage , ImageCaptureExtenderImpl.getCaptureStages ).
  • Add processors for preview and still capture that's capable of processing YUV_420_888 stream.

Let's see how Camera2/X invokes the extensions-interface to achieve the three app flows mentioned above.

App flow 1: Check extension availability

BasicExtenderAppFlow1

Figure 3. App flow 1 on Basic Extender

In this flow, Camera2/X directly calls the isExtensionAvailable() method of both PreviewExtenderImpl and ImageCaptureExtenderImpl without calling init() . Both Extender classes must return true to enable the extensions.

This is often the first step for apps to check if the given extension type is supported for a given camera ID before enabling the extension. This is because some extensions are supported only on certain camera IDs.

App flow 2: Query information

BasicExtenderAppFlow2

Figure 4. App flow 2 on Basic Extender

After determining if the extension is available, apps should query the following information before enabling the extension.

  • Still capture latency range: ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange returns the range of the capture latency for the app to evaluate if it's appropriate to enable the extension for the current scenario.

  • Supported sizes for the preview and capture surface: ImageCaptureExtenderImpl.getSupportedResolutions and PreviewExtenderImpl.getSupportedResolutions return a list of image formats and the sizes that are supported for surface format and size.

  • Supported request and result keys: Camera2/X invokes the following methods to retrieve the supported capture request keys and result keys from your implementation:

    • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys
    • ImageCaptureExtenderImpl.getAvailableCapturetResultKeys

Camera2/X always calls init() first on these Extender classes before querying for more information.

App flow 3: Preview/still capture with extension enabled (HAL implementation)

BasicExtenderAppFlow3

Figure 5. App flow 3 on Basic Extender

The above diagram illustrates the main flow of enabling preview and still capture with an extension without any processor. This means the camera HAL processes the extension.

In this flow, Camera2/X first calls init() then onInit , which notifies you that a camera session is about to start with the specified extensions. You can do heavy-lifting initialization in onInit() .

When configuring CameraCaptureSession , Camera2/X invokes onPresetSession to get the session parameters. After the capture session is configured successfully, Camera2/X invokes onEnableSession returning a CaptureStageImpl instance that contains the capture parameters. Camera2/X immediately sends a single request with these capture parameters to notify the HAL. Similarly, before the capture session is closed, Camera2/X invokes onDisableSession and then sends a single request with the returned capture parameters.

The repeating request triggered by Camera2/X contains the request parameters returned by PreviewExtenderImpl.getCaptureStage() . Furthermore, the still capture request contains the parameters returned by ImageCaptureExtenderImpl.getCaptureStages() .

Finally, Camera2/X invokes onDeInit() after the camera session has finished. You can release resources in onDeinit() .

Preview processor

In addition to the camera HAL, you can also implement extensions in a processor.

Implement PreviewExtenderImpl.getProcessorType to specify the processor type as explained below:

  • PROCESSOR_TYPE_NONE : No processor. Images are processed in the camera HAL.

  • PROCESSOR_TYPE_REQUEST_UPDATE_ONLY : The processor type lets you update the repeating request with new capture request parameters based on the latest TotalCaptureResult .

    PreviewExtenderImpl.getProcessor must return a RequestUpdateProcessorImpl instance that processes the TotalCaptureResult instance and returns a CaptureStageImpl instance to update the repeating request. PreviewExtenderImpl.getCaptureStage() should also reflect the result of the processing and return the latest CaptureStageImpl .

  • PROCESSOR_TYPE_IMAGE_PROCESSOR : This type allows you to implement a processor to process YUV_420_888 images and write the output to a PRIVATE surface.

    You need to implement and return a PreviewImageProcessorImpl instance in PreviewExtenderImpl.getProcessor . The processor is responsible for processing YUV_420_888 input images. It should write the output to the PRIVATE format of preview. Camera2/X uses a YUV_420_888 surface instead of PRIVATE to configure the CameraCaptureSession for preview.

    See following illustration for the flow:

PreviewProcessor

Figure 6. Preview flow with PreviewImageProcessorImpl

The PreviewImageProcessorImpl interface extends ProcessImpl and has three important methods:

  • onOutputSurface(Surface surface, int imageFormat) sets the output surface for the processor. For PreviewImageProcessorImpl , imageFormat is a pixel format such as PixelFormat.RGBA_8888 .

  • onResolutionUpdate(Size size) sets the size of the input image.

  • onImageFormatUpdate(int imageFormat) sets the image format of the input image. Currently, it can only be YUV_420_888 .

Image capture processor

For still capture, you can implement a processor by returning a CaptureProcessorImpl instance using ImageCaptureExtenderImpl.getCaptureProcessor . The processor is responsible to process a list of captured YUV_420_888 images and TotalCaptureResult instances and write the output to a YUV_420_888 surface.

You can safely assume that preview is enabled and running before sending the still capture request.

See the flow in the diagram below:

CaptureProcessor

Figure 7. Still capture flow with CaptureProcessorImpl

  1. Camera2/X uses a YUV_420_888 format surface for still capture to configure the capture session. Camera2/X prepares CaptureProcessorImpl by calling:

    • CaptureProcessorImpl.onImageFormatUpdate() with YUV_420_888 .
    • CaptureProcessorImpl.onResolutionUpdate() with the input image size.
    • CaptureProcessorImpl.onOutputSurface() with an output YUV_420_888 surface.

  2. ImageCaptureExtenderImpl.getCaptureStages returns a list of CaptureStageImpl , where each element maps to a CaptureRequest instance with capture parameters that are sent by Camera2/X. For example, if it returns a list of three CaptureStageImpl instances, Camera2/X sends three capture requests with corresponding capture parameters using the captureBurst API.

  3. The received images and TotalCaptureResult instances are bundled together and sent to CaptureProcessorImpl for processing.

  4. CaptureProcessorImpl writes the result Image ( YUV_420_888 format) to the output surface specified by the onOutputSurface() call. Camera2/X converts it into JPEG images if necessary.

Support capture request keys and results

In addition to camera preview and capture, apps can set zoom, flash parameters, or trigger a tap-to-focus. These parameters might not be compatible with your extension implementation.

The following methods have been added to extensions-interface 1.3.0 to allow you to expose the parameters that your implementation supports:

  • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys() returns the capture request keys supported by your implementation.
  • ImageCaptureExtenderImpl.getAvailableCaptureResultKeys() returns the capture result keys that are contained in the capture result.

If the camera HAL processes the extension, Camera2/X retrieves the capture results in CameraCaptureSession.CaptureCallback . However, if the processor is implemented, then Camera2/X retrieves the capture results in ProcessResultImpl , which is passed to the process() method in PreviewImageProcessorImpl and CaptureProcessorImpl . You're responsible for reporting the capture result through ProcessResultImpl to Camera2/X.

See the definition of the CaptureProcessorImpl interface below as an example. In extensions-interface 1.3.0 or higher, the second process() call is invoked:

Interface CaptureProcessorImpl extends ProcessorImpl {
    // invoked when extensions-interface version < 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results);
    // invoked when extensions-interface version >= 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results,
            ProcessResultImpl resultCallback, Executor executor);
}

For common camera operations like zoom, tap-to-focus, flash, and exposure compensation, we recommend supporting the following keys for both capture request and capture result:

  • Zoom:
    • CaptureRequest#CONTROL_ZOOM_RATIO
    • CaptureRequest#SCALER_CROP_REGION
  • Tap-to-focus:
    • CaptureRequest#CONTROL_AF_MODE
    • CaptureRequest#CONTROL_AF_TRIGGER
    • CaptureRequest#CONTROL_AF_REGIONS
    • CaptureRequest#CONTROL_AE_REGIONS
    • CaptureRequest#CONTROL_AWB_REGIONS
  • Flash:
    • CaptureRequest#CONTROL_AE_MODE
    • CaptureRequest#CONTROL_AE_PRECAPTURE_TRIGGER
    • CaptureRequest#FLASH_MODE
  • Exposure compensation:
    • CaptureRequest#CONTROL_AE_EXPOSURE_COMPENSATION

For Basic Extenders that implement 1.2.0 or prior versions, the CameraX Extensions API explicitly supports all the above keys. For extensions-interface 1.3.0, both CameraX and Camera2 honor the returned list and support only the keys contained in it. For example, if you decide to return only CaptureRequest#CONTROL_ZOOM_RATIO and CaptureRequest#SCALER_CROP_REGION in the 1.3.0 implementation, then that means only zoom is supported for the app while tap-to-focus, flash, and exposure compensation aren't allowed.

Advanced Extender

Advanced Extender is a type of vendor implementation based on the Camera2 API. This Extender type was added in extensions-interface 1.2.0. Depending on the device manufacturer, extensions might be implemented in the app layer, which depends on the following factors:

  • Custom stream configuration: Configure custom streams like RAW stream or have multiple streams for different physical camera IDs.

  • Capability to send Camera2 requests: Support a complicated interaction logic that can send capture requests with parameters based on the results of previous requests.

Advanced Extender provides a wrapper, or an intermediate layer, so you can customize the stream configuration and send capture requests on demand.

Files to implement

To switch to the Advanced Extender implementation, the isAdvancedExtenderImplemented() method in ExtensionVersionImpl must return true . For each extension type, OEMs must implement the corresponding Extender classes. The Advanced Extender implementation files are in the advanced package.

Extender classes to implement
Night advanced/NightAdvancedExtenderImpl.java
HDR advanced/HdrAdvancedExtenderImpl.java
Auto advanced/AutoAdvancedExtenderImpl.java
Bokeh advanced/BokehAdvancedExtenderImpl.java
Face Retouch advanced/BeautyAdvancedExtenderImpl.java

We use AdvancedExtenderImpl as a placeholder in the following example. Replace it with the name of the Extender file for the extension you're implementing.

Let's see how Camera2/X invokes the extensions-interface to achieve the three app flows.

App flow 1: Check extensions availability

AdvancedAppFlow1

Figure 8. App flow 1 on Advanced Extender

First, the app checks if the given extension is supported.

App flow 2: Query information

AdvancedAppFlow2

Figure 9. App flow 2 on Advanced Extender

After calling AdvancedExtenderImpl.init() , the app can query the following the information on AdvancedExtenderImpl :

  • Estimated still capture latency: AdvancedExtenderImpl.getEstimatedCaptureLatencyRange() returns the range of the capture latency for the app to evaluate if it is appropriate to enable the extension for the current scenario.

  • Supported resolutions for preview and still capture:

    • AdvancedExtenderImpl.getSupportedPreviewOutputResolutions() returns a map of image format to the sizes list that are supported for preview surface format and size. OEMs must support at least the PRIVATE format.

    • AdvancedExtenderImpl.getSupportedCaptureOutputResolutions() returns the supported format and sizes for still capture surface. OEMs must support both JPEG and YUV_420_888 format output.

    • AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() returns the supported sizes for an extra YUV_420_888 stream for image analysis. If the image analysis YUV surface isn't supported, getSupportedYuvAnalysisResolutions() should return null or an empty list.

  • Available capture request keys/results (added in extensions-interface 1.3.0): Camera2/X invokes the following methods to retrieve the supported capture request keys and result keys from your implementation:

    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys
    • AdvancedExtenderImpl.getAvailableCaptureResultKeys

For more information, see Support capture request keys and results .

App flow 3: Preview/still capture with extension enabled

AdvancedAppFlow3

Figure 10. App flow 3 on Advanced Extender

The above diagram shows the main flow for starting preview and still capture for the Advanced Extender type. Let's walk through each step.

  1. SessionProcessorImpl instance

    The core Advanced Extender implementation is in SessionProcessorImpl , which is responsible for providing customized session configuration and sending capture requests to initiate the preview and still capture request. AdvancedExtenderImpl.createSessionProcessor() is invoked to return the SessionProcessorImpl instance.

  2. initSession

    SessionProcessorImpl.initSession() initializes the session for the extension. This is where you allocate resources and return a session configuration for preparing a CameraCaptureSession .

    For the input parameters, Camera2/X specifies the output surface configurations for preview, still capture, and an optional YUV image analysis. This output surface configuration ( OutputSurfaceImpl ) contains the surface, size and image format that are retrieved by following methods in AdvancedExtenderImpl :

    • getSupportedPreviewOutputResolutions()
    • getSupportedCaptureOutputResolutions()
    • getSupportedYuvAnalysisResolutions()

    You must return a Camera2SessionConfigImpl instance, which consists of a list of Camera2OutputConfigImpl instances and the session parameters used for configuring CameraCaptureSession . You're responsible for outputting the correct camera images to the output surfaces passed in by Camera2/X. Here are some options to enable the output:

    • Processing in camera HAL: You can directly add the output surfaces to CameraCaptureSession with a SurfaceOutputConfigImpl implementation. This configures the supplied output surface to the camera pipeline and allows the camera HAL to process the image.

    • Processing intermediate ImageReader surface (RAW, YUV, etc): Add the intermediate ImageReader surfaces to the CameraCaptureSession with an ImageReaderOutputConfigImpl instance.

      You need to process the intermediate images and write the result image to the output surface.

    • Use Camera2 surface sharing: Use surface sharing with another surface by adding any Camera2OutputConfigImpl instance to the getSurfaceSharingOutputConfigs() method of another Camera2OutputConfigImpl instance. The surface format and size must be identical.

    All Camera2OutputConfigImpl including SurfaceOutputConfigImpl and ImageReaderOutputConfigImpl must have a unique ID ( getId() ), which is used to specify the target surface and retrieve the image from ImageReaderOutputConfigImpl .

  3. onCaptureSessionStart and RequestProcessorImpl

    When CameraCaptureSession starts and the Camera framework invokes onConfigured() , then Camera2/X invokes SessionProcessorImpl.onCaptureSessionStart() with the Camera2 request wrapper RequestProcessImpl . Camera2/X implements RequestProcessImpl , which enables you to execute the capture requests , and retrieve images if ImageReaderOutputConfigImpl is used.

    The RequestProcessImpl APIs are similar to the Camera2 CameraCaptureSession APIs in terms of executing requests. The differences are:

    • The target surface is specified by the ID of the Camera2OutputConfigImpl instance.
    • The capability of retrieving the image of the ImageReader .

    You can call RequestProcessorImpl.setImageProcessor() with a specified Camera2OutputConfigImpl ID to register an ImageProcessorImpl instance to receive images.

    The RequestProcessImpl instance becomes invalid after Camera2/X calls SessionProcessorImpl.onCaptureSessionEnd() .

  4. Start the preview and take a picture

    In the Advanced Extender implementation, you can send capture requests through the RequestProcessorImpl interface. Camera2/X notifies you to start the repeating request for preview or the still capture sequence by calling SessionProcessorImpl#startRepeating and SessionProcessorImpl#startCapture respectively. You should send capture requests to satisfy these preview and still-capture requests.

    Camera2/X also sets the capture request parameters through SessionProcessorImpl#setParameters . You must set these request parameters (if parameters are supported) on both the repeating and single requests.

    You must support at least CaptureRequest.JPEG_ORIENTATION and CaptureRequest.JPEG_QUALITY . extensions-interface 1.3.0 supports request and result keys, which are exposed by the following methods:

    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys()
    • AdvancedExtenderImpl.getAvailableCaptureResultKeys()

    When developers set the keys in the getAvailableCaptureRequestKeys list, you must enable the parameters and ensure the capture result contains the keys in the getAvailableCaptureResultKeys list.

  5. startTrigger

    SessionProcessorImpl.startTrigger() is invoked to start the trigger such as CaptureRequest.CONTROL_AF_TRIGGER and CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER . You can disregard any capture request keys that weren't advertised in AdvancedExtenderImpl.getAvailableCaptureRequestKeys() .

    startTrigger() has been supported since extensions-interface 1.3.0. It enables apps to implement tap-to-focus and flash with extensions.

  6. Clean up

    When finishing a capture session, SessionProcessorImpl.onCaptureSessionEnd() is invoked ahead of closing CameraCaptureSession . After the capture session has closed, deInitSession() performs the clean up.

Support preview, still capture, and image analysis

You should apply the extension for both the preview and still capture use cases. However, if the latency is too high to smoothly show the preview, you can apply the extension only for still capture.

For the Basic Extender type, regardless of enabling the extension for preview, you must implement both ImageCaptureExtenderImpl and PreviewExtenderImpl for a given extension. Often, an app also uses a YUV stream to analyze the image content such as finding QR codes or text. To better support this use case , you should support the stream combination of preview, still capture, and a YUV_420_888 stream for configuring CameraCaptureSession . This means that if you implement a processor, then you have to support the stream combination of three YUV_420_888 streams.

For Advanced Extender, Camera2/X passes three output surfaces to the SessionProcessorImpl.initSession() call. These output surfaces are for preview , still capture, and image analysis, respectively. You must ensure that preview and still capture output surfaces show the valid output. However, for the image analysis output surface, ensure it's working only when it's non-null. If your implementation can't support the image analysis stream, you can return an empty list in AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() . This ensures the image analysis output surface is always null in SessionProcessorImpl.initSession() .

Support video capture

The current Camera Extension architecture supports only the preview and still capture use cases. We don't support enabling the extension on the MediaCodec or MediaRecorder surfaces for recording the video. However, it's possible for apps to record the preview output.

Supporting MediaCodec and MediaRecorder surfaces is under investigation.

Extensions interface version history

The following table shows the Camera Extension interface version history. You should always implement the vendor library with the latest version.

Version Added features
1.0.0
  • Version verification
    • ExtensionVersionImpl
  • Basic Extender
    • PreviewExtenderImpl
    • ImageCaptureExtenderImpl
    • Processor
      • PreviewImageProcessorImpl
      • CaptureProcessorImpl
      • RequestUpdateProcessorImpl
1.1.0
  • Library initialization
    • InitializerImpl
  • Expose supported resolutions
    • PreviewExtenderImpl.getSupportedResolutions
    • ImageCaptureExtenderImpl.getSupportedResolutions
1.2.0
  • AdvancedExtender
    • AdvancedExtenderImpl
    • SessionProcessorImpl
  • Get estimated capture latency
    • ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange
1.3.0
  • Expose supported capture request keys/results keys
    • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys
    • AdvancedExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys
    • New process() call that takes ProcessResultImpl in PreviewImageProcessorImpl and CaptureProcessorImpl
    • Support trigger type request
      • AdvancedExtenderImpl.startTrigger

Reference implementation

The following reference OEM vendor library implementations are available in frameworks/ex .

  • advancedSample : A basic implementation of Advanced Extender.

  • sample : A basic implementation of Basic Extender.

  • service_based_sample : An implementation that demonstrates how to host Camera Extensions in a Service . This implementation contains the following components:

    • oem_library : A Camera Extensions OEM library for Camera2 and CameraX Extensions APIs that implements Extensions-Interface . This acts as a passthrough that forwards calls from Extensions-Interface to the service. This library also provides AIDL files and wrapper classes to communicate with the service.

      Advanced Extender is enabled by default. To enable the Basic Extender, change ExtensionsVersionImpl#isAdvancedExtenderImplemented to return false .

    • extensions_service : A sample implementation of the Extensions Service. Add your implementation here. The interface to implement in the service is similar to the Extensions-Interface . For example, implementing the IAdvancedExtenderImpl.Stub performs the same operations as AdvancedExtenderImpl . ImageWrapper and TotalCaptureResultWrapper are required to make Image and TotalCaptureResult parcelable.

Set up the vendor library on a device

The OEM vendor library isn't built into an app; it's loaded from the device at runtime by Camera2/X. In CameraX, the <uses-library> tag declares that the androidx.camera.extensions.impl library, which is defined in the AndroidManifest.xml file of the camera-extensions library, is a dependency of CameraX and must be loaded at runtime. In Camera2, the framework loads an extensions service that also declares that the <uses-library> loads the same androidx.camera.extensions.impl library at runtime.

This allows third-party apps using extensions to automatically load the OEM vendor library. The OEM library is marked as optional so apps can run on devices that don't have the library on the device. Camera2/X handles this behavior automatically when an app tries to use a camera extension as long as the device manufacturer places the OEM library on the device so that it can be discovered by the app.

To set up the OEM library on a device, do the following:

  1. Add a permission file, which is required by the <uses-library> tag, using the following format: /etc/permissions/ ANY_FILENAME .xml . For example, /etc/permissions/camera_extensions.xml . The files in this directory provide a mapping of the library named in <uses-library> to the actual file path on the device.

  2. Use the example below to add the required information to the file.

    • name must be androidx.camera.extensions.impl as that's the library that CameraX searches for.
    • file is the absolute path of the file that contains the extensions implementation (for example, /system/framework/androidx.camera.extensions.impl.jar ).
    <?xml version="1.0" encoding="utf-8"?>
<permissions>
    <library name="androidx.camera.extensions.impl"
             file="OEM_IMPLEMENTED_JAR" />
</permissions>

In Android 12 or higher, devices supporting CameraX extensions must have the ro.camerax.extensions.enabled property set to true , which allows for querying whether a device supports extensions. To do this, add the following line in the device make file:

PRODUCT_VENDOR_PROPERTIES += \
    ro.camerax.extensions.enabled=true \

Validation

To test your implementation of the OEM vendor library during the development stage, use the example app at androidx-main/camera/integration-tests/extensionstestapp/ , which runs through various vendor extensions.

After you complete your implementation, use the Camera Extensions Validation Tool to run automated and manual tests to verify that the vendor library is implemented correctly.

Extended scene mode versus Camera Extensions

For the bokeh extension, in addition to exposing it using Camera Extensions, you can expose the extension using the extended scene mode, which is enabled through the CONTROL_EXTENDED_SCENE_MODE key. For more implementation details, see Camera Bokeh .

Extended scene mode has fewer restrictions compared to Camera Extensions for camera2 apps. For example, you can enable extended scene mode in a regular CameraCaptureSession instance that supports flexible stream combinations and capture request parameters. In contrast, camera extensions support only a fixed set of stream types and have limited support for capture request parameters.

A downside of extended scene mode is that you can only implement it in the camera HAL, which means that it must be verified to work across all orthogonal controls available to app developers.

We recommend exposing bokeh using both the extended scene mode and Camera Extensions because apps might prefer to use a particular API to enable bokeh. We recommend first using the extended scene mode because this is the most flexible way for apps to enable the bokeh extension. Then you can implement the camera extensions interface based on the extended scene mode. If implementing bokeh in the camera HAL is difficult, for example, because it requires a post processor running in the app layer to process images, we recommend implementing the bokeh extension using the Camera Extensions interface.

Frequently asked questions (FAQs)

Are there any restrictions on API levels?

Yes. This depends on the Android API feature set that's required by the OEM vendor library implementation. For example, ExtenderStateListener.onPresetSession() uses the SessionConfiguration.setSessionParameters() call to set a baseline set of tags. This call is available only on API level 28 and higher. For details on specific interface methods, see the API reference documentation .