Understanding the Android graphics system starts behind the scenes with BufferQueue and the gralloc HAL.
The BufferQueue class is at the heart of everything graphical in Android. Its role is simple: Connect something that generates buffers of graphical data (the producer) to something that accepts the data for display or further processing (the consumer). Nearly everything that moves buffers of graphical data through the system relies on BufferQueue.
The gralloc memory allocator performs buffer allocations and is
implemented through a vendor-specific HAL interface (see
alloc() function takes expected arguments (width, height, pixel
format) as well as a set of usage flags (detailed below).
BufferQueue producers and consumers
Basic usage is straightforward: The producer requests a free buffer
dequeueBuffer()), specifying a set of characteristics including
width, height, pixel format, and usage flags. The producer populates the buffer
and returns it to the queue (
queueBuffer()). Later, the consumer
acquires the buffer (
acquireBuffer()) and makes use of the buffer
contents. When the consumer is done, it returns the buffer to the queue
Recent Android devices support the sync framework, which enables the system to do nifty things when combined with hardware components that can manipulate graphics data asynchronously. For example, a producer can submit a series of OpenGL ES drawing commands and then enqueue the output buffer before rendering completes. The buffer is accompanied by a fence that signals when the contents are ready. A second fence accompanies the buffer when it is returned to the free list, so the consumer can release the buffer while the contents are still in use. This approach improves latency and throughput as the buffers move through the system.
Some characteristics of the queue, such as the maximum number of buffers it can hold, are determined jointly by the producer and the consumer. However, the BufferQueue is responsible for allocating buffers as it needs them. Buffers are retained unless the characteristics change; for example, if the producer requests buffers with a different size, old buffers are freed and new buffers are allocated on demand.
Producers and consumers can live in different processes. Currently, the consumer always creates and owns the data structure. In older versions of Android, only the producer side was binderized (i.e. producer could be in a remote process but consumer had to live in the process where the queue was created). Android 4.4 and later releases moved toward a more general implementation.
Buffer contents are never copied by BufferQueue (moving that much data around would be very inefficient). Instead, buffers are always passed by handle.
gralloc HAL usage flags
The gralloc allocator is not just another way to allocate memory on the native heap; in some situations, the allocated memory may not be cache-coherent or could be totally inaccessible from user space. The nature of the allocation is determined by the usage flags, which include attributes such as:
- How often the memory will be accessed from software (CPU)
- How often the memory will be accessed from hardware (GPU)
- Whether the memory will be used as an OpenGL ES (GLES) texture
- Whether the memory will be used by a video encoder
For example, if your format specifies RGBA 8888 pixels, and you indicate the buffer will be accessed from software (meaning your application will touch pixels directly) then the allocator must create a buffer with 4 bytes per pixel in R-G-B-A order. If instead, you say the buffer will be only accessed from hardware and as a GLES texture, the allocator can do anything the GLES driver wants—BGRA ordering, non-linear swizzled layouts, alternative color formats, etc. Allowing the hardware to use its preferred format can improve performance.
Some values cannot be combined on certain platforms. For example, the video encoder flag may require YUV pixels, so adding software access and specifying RGBA 8888 would fail.
The handle returned by the gralloc allocator can be passed between processes through Binder.
Tracking BufferQueue with systrace
To really understand how graphics buffers move around, use systrace. The system-level graphics code is well instrumented, as is much of the relevant app framework code.
A full description of how to use systrace effectively would fill a rather
long document. Start by enabling the
sched tags. You'll also see BufferQueues in the trace. If you've
used systrace before, you've probably seen them but maybe weren't sure what they
were. As an example, if you grab a trace while
Grafika's "Play video
(SurfaceView)" is running, the row labeled SurfaceView tells you how
many buffers were queued up at any given time.
The value increments while the app is active—triggering the rendering of frames by the MediaCodec decoder—and decrements while SurfaceFlinger is doing work, consuming buffers. When showing video at 30fps, the queue's value varies from 0 to 1 because the ~60fps display can easily keep up with the source. (Notice also that SurfaceFlinger only wakes when there's work to be done, not 60 times per second. The system tries very hard to avoid work and will disable VSYNC entirely if nothing is updating the screen.)
If you switch to Grafika's "Play video (TextureView)" and grab a new trace, you'll see a row labeled com.android.grafika/com.android.grafika.PlayMovieActivity. This is the main UI layer, which is just another BufferQueue. Because TextureView renders into the UI layer (rather than a separate layer), you'll see all of the video-driven updates here.
For more information about the systrace tool, refer to Systrace documentation.