You can extend the media extractor and media codec components using vendor
extensions. The MediaSession2 and MediaParser APIs can't be customized (but you
can upstream changes for the legacy MediaPlayer and MediaSession APIs).
To support additional media types in the Android media framework, you need to create a custom extractor and decoder. For example, to add support for Windows Media video in AVI files, you need to create an AVI Extractor and a Windows Media video Decoder.
About extensions
If the default media extractors don't meet your requirements, you can place
custom extractor plugins in /system/lib[64]/extractors/. The extractor process
automatically loads extractor plugins from the Google-provided APEX package and
from /system/lib[64]/extractors/.
Similarly, you can set up custom media codec services that use the Codec 2.0
interface defined in frameworks/av/media/codec2/core/. For a basic
implementation, refer to frameworks/av/media/codec2/hidl/services/. The
library entry point is the C2ComponentStore interface. For an example, refer
to the default software codec store implementation at
frameworks/av/media/codec2/vndk/C2Store.cpp.
When using your own APEX, structure the codec service and load the APEX file
using the same processes as the mediaswcodec service. To do so, define a
top-level shared library responsible for registering all C2 components, then
create an APEX package (with transitive dependencies) that resides in the vendor
partition. When the vendor codec service process starts, it can then load this
top-level entry point.
Create an extractor
When adding an extractor for a new format, ensure the extractor depends only on
stable NDK APIs and doesn't depend on any private APIs. Extractors should
implement the API defined by
frameworks/av/include/media/MediaExtractorPluginApi.h and can use the C++
convenience wrappers in
frameworks/av/include/media/MediaExtractorPluginHelper.h. Because Android
10 or higher supports only the highest version of the
extractor API, be sure to model your extractor after the extractor with the
highest API version number.
Place custom extractors in /system/lib/64/extractors or a vendor APEX,
which is opened along with the Google APEX containing the Google extractors. To
verify the framework loaded your extractor, run the following command.
adb shell dumpsys media.extractor
You should get a list of available extractors that is similar to the following.
Available extractors:
AAC Extractor: plugin\_version(2), uuid(4fd80eae03d24d729eb948fa6bb54613), version(1), path(/system/lib64/extractors/libaacextractor.so)
AMR Extractor: plugin\_version(2), uuid(c86639c92f3140aca715fa01b4493aaf), version(1), path(/system/lib64/extractors/libamrextractor.so)
FLAC Extractor: plugin\_version(2), uuid(1364b048cc454fda9934327d0ebf9829), version(1), path(/system/lib64/extractors/libflacextractor.so)
MIDI Extractor: plugin\_version(2), uuid(ef6cca0af8a243e6ba5fdfcd7c9a7ef2), version(1), path(/system/lib64/extractors/libmidiextractor.so)
MP3 Extractor: plugin\_version(2), uuid(812a3f6cc8cf46deb5293774b14103d4), version(1), path(/system/lib64/extractors/libmp3extractor.so)
MP4 Extractor: plugin\_version(2), uuid(27575c6744174c548d3d8e626985a164), version(2), path(/system/lib64/extractors/libmp4extractor.so)
MPEG2-PS/TS Extractor: plugin\_version(1), uuid(3d1dcfebe40a436da574c2438a555e5f), version(1), path(/system/lib64/extractors/libmpeg2extractor.so)
Matroska Extractor: plugin\_version(2), uuid(abbedd9238c44904a4c1b3f45f899980), version(1), path(/system/lib64/extractors/libmkvextractor.so)
Ogg Extractor: plugin\_version(2), uuid(8cc5cd06f772495e8a62cba9649374e9), version(1), path(/system/lib64/extractors/liboggextractor.so)
WAV Extractor: plugin\_version(3), uuid(7d61385858374a3884c5332d1cddee27), version(1), path(/system/lib64/extractors/libwavextractor.so)
If your custom extractor supports a format that is already supported by a
Google-provided extractor, you can force the framework to use your extractor by
using the Sniff() function to return a higher confidence level than the
Google-provided one.
When the media framework loads your extractor (from /system/lib/64/extractors
or from a vendor APEX), it recognizes the file and gets information about its
content. The next step is to add a decoder for the format so the framework can
understand how to parse the file content.
Create a custom decoder
You need a custom decoder for any format that isn't already supported by a Google-provided decoder. For example:
To add media framework support for AVI files containing MP3, you need an AVI extractor but don't need a MP3 decoder because one already exists.
To add media framework support for AVI files containing Windows Media, you need both an AVI extractor and a Windows Media decoder.
Adding a new decoder is similar to adding your own hardware decoders for AVC or HEVC.
While the extractor publishes the MIME type of the media tracks that it
contains, the codecs that support these MIME types need to be present for the
file to be fully supported. The actual MIME type string used is strictly an
agreement between the extractor and the codec (the string doesn't need to be
added to the MediaDefs.h file).
Integrate with media scanner
The media scanner looks for new file types and adds them to the media database.
To have the media scanner handle your custom file type, the scanner needs to
know about it. In Android 10 or higher, MimeUtils (in
libcore) maintains the MIME-to-extension mapping. Previously, this mapping was
handled in the MediaFile.java file, which continues to contain a mapping from
MIME type to MTP format constants.
An extractor can export a list of filename extensions they support (such as MP3
or MP4). However, only LegacyMediaScanner uses that; it has no effect on
ModernMediaScanner, which used by default.