Customize media components

You can extend the media extractor and media codec components using vendor extensions. The MediaSession2 and MediaParser APIs can't be customized (but you can upstream changes for the legacy MediaPlayer and MediaSession APIs).

To support additional media types in the Android media framework, you need to create a custom extractor and decoder. For example, to add support for Windows Media video in AVI files, you need to create an AVI Extractor and a Windows Media video Decoder.

About extensions

If the default media extractors don't meet your requirements, you can place custom extractor plugins in /system/lib[64]/extractors/. The extractor process automatically loads extractor plugins from the Google-provided APEX package and from /system/lib[64]/extractors/.

Similarly, you can set up custom media codec services that use the Codec 2.0 interface defined in frameworks/av/media/codec2/core/. For a basic implementation, refer to frameworks/av/media/codec2/hidl/services/. The library entry point is the C2ComponentStore interface. For an example, refer to the default software codec store implementation at frameworks/av/media/codec2/vndk/C2Store.cpp.

When using your own APEX, structure the codec service and load the APEX file using the same processes as the mediaswcodec service. To do so, define a top-level shared library responsible for registering all C2 components, then create an APEX package (with transitive dependencies) that resides in the vendor partition. When the vendor codec service process starts, it can then load this top-level entry point.

Create an extractor

When adding an extractor for a new format, ensure the extractor depends only on stable NDK APIs and doesn't depend on any private APIs. Extractors should implement the API defined by frameworks/av/include/media/MediaExtractorPluginApi.h and can use the C++ convenience wrappers in frameworks/av/include/media/MediaExtractorPluginHelper.h. Because Android 10 or higher supports only the highest version of the extractor API, be sure to model your extractor after the extractor with the highest API version number.

Place custom extractors in /system/lib/64/extractors or a vendor APEX, which is opened along with the Google APEX containing the Google extractors. To verify the framework loaded your extractor, run the following command.

adb shell dumpsys media.extractor

You should get a list of available extractors that is similar to the following.

Available extractors:
AAC Extractor: plugin\_version(2), uuid(4fd80eae03d24d729eb948fa6bb54613), version(1), path(/system/lib64/extractors/libaacextractor.so)
AMR Extractor: plugin\_version(2), uuid(c86639c92f3140aca715fa01b4493aaf), version(1), path(/system/lib64/extractors/libamrextractor.so)
FLAC Extractor: plugin\_version(2), uuid(1364b048cc454fda9934327d0ebf9829), version(1), path(/system/lib64/extractors/libflacextractor.so)
MIDI Extractor: plugin\_version(2), uuid(ef6cca0af8a243e6ba5fdfcd7c9a7ef2), version(1), path(/system/lib64/extractors/libmidiextractor.so)
MP3 Extractor: plugin\_version(2), uuid(812a3f6cc8cf46deb5293774b14103d4), version(1), path(/system/lib64/extractors/libmp3extractor.so)
MP4 Extractor: plugin\_version(2), uuid(27575c6744174c548d3d8e626985a164), version(2), path(/system/lib64/extractors/libmp4extractor.so)
MPEG2-PS/TS Extractor: plugin\_version(1), uuid(3d1dcfebe40a436da574c2438a555e5f), version(1), path(/system/lib64/extractors/libmpeg2extractor.so)
Matroska Extractor: plugin\_version(2), uuid(abbedd9238c44904a4c1b3f45f899980), version(1), path(/system/lib64/extractors/libmkvextractor.so)
Ogg Extractor: plugin\_version(2), uuid(8cc5cd06f772495e8a62cba9649374e9), version(1), path(/system/lib64/extractors/liboggextractor.so)
WAV Extractor: plugin\_version(3), uuid(7d61385858374a3884c5332d1cddee27), version(1), path(/system/lib64/extractors/libwavextractor.so)

If your custom extractor supports a format that is already supported by a Google-provided extractor, you can force the framework to use your extractor by using the Sniff() function to return a higher confidence level than the Google-provided one.

When the media framework loads your extractor (from /system/lib/64/extractors or from a vendor APEX), it recognizes the file and gets information about its content. The next step is to add a decoder for the format so the framework can understand how to parse the file content.

Create a custom decoder

You need a custom decoder for any format that isn't already supported by a Google-provided decoder. For example:

  • To add media framework support for AVI files containing MP3, you need an AVI extractor but don't need a MP3 decoder because one already exists.

  • To add media framework support for AVI files containing Windows Media, you need both an AVI extractor and a Windows Media decoder.

Adding a new decoder is similar to adding your own hardware decoders for AVC or HEVC.

While the extractor publishes the MIME type of the media tracks that it contains, the codecs that support these MIME types need to be present for the file to be fully supported. The actual MIME type string used is strictly an agreement between the extractor and the codec (the string doesn't need to be added to the MediaDefs.h file).

Integrate with media scanner

The media scanner looks for new file types and adds them to the media database. To have the media scanner handle your custom file type, the scanner needs to know about it. In Android 10 or higher, MimeUtils (in libcore) maintains the MIME-to-extension mapping. Previously, this mapping was handled in the MediaFile.java file, which continues to contain a mapping from MIME type to MTP format constants.

An extractor can export a list of filename extensions they support (such as MP3 or MP4). However, only LegacyMediaScanner uses that; it has no effect on ModernMediaScanner, which used by default.