BufferQueue and Gralloc

The BufferQueue class connects components that generate buffers of graphical data (producers) to components that accept the data for display or further processing (consumers). Nearly everything that moves buffers of graphical data through the system relies on BufferQueue.

The Gralloc memory allocator performs buffer allocations and is implemented through two vendor-specific HIDL interfaces (see hardware/interfaces/graphics/allocator/ and hardware/interfaces/graphics/mapper/). The allocate() function takes expected arguments (width, height, pixel format) as well as a set of usage flags.

BufferQueue producers and consumers

Consumers create and own the BufferQueue data structure and can exist in different processes than their producers. When a producer needs a buffer, it requests a free buffer from BufferQueue by calling dequeueBuffer(), specifying the buffers's width, height, pixel format, and usage flags. The producer then populates the buffer and returns the buffer to the queue by calling queueBuffer(). Next, the consumer acquires the buffer with acquireBuffer() and makes use of the buffer contents. When the consumer is done, it returns the buffer to the queue by calling releaseBuffer(). The sync framework controls how buffers move through the Android graphics pipeline.

Some characteristics of the BufferQueue, such as the maximum number of buffers it can hold, are determined jointly by the producer and the consumer. However, the BufferQueue allocates buffers as it needs them. Buffers are retained unless the characteristics change; for example, if a producer requests buffers with a different size, old buffers are freed and new buffers are allocated on demand.

Buffer contents are never copied by BufferQueue, as moving that much data around is inefficient. Instead, buffers are always passed by a handle.

Track BufferQueue with Systrace

To understand how graphics buffers move around, use Systrace, a tool that records device activity over a short period of time. The system-level graphics code is well instrumented, as is much of the relevant app framework code.

To use Systrace, enable the gfx, view, and sched tags. BufferQueue objects are displayed in the trace. As an example, if you take a trace while Grafika's Play video (SurfaceView) is running, the row labeled SurfaceView tells you how many buffers were queued up at any given time.

The value increments while the app is active, which triggers the rendering of frames by the MediaCodec decoder. The value decrements while SurfaceFlinger is working and consuming buffers. When showing video at 30 fps, the queue's value varies from 0 to 1 because the ~60 fps display can keep up with the source. SurfaceFlinger wakes only when there's work to be done, not 60 times per second. The system tries to avoid work and disables VSYNC if nothing is updating the screen.

If you switch to Grafika's Play video (TextureView) and grab a new trace, you see a row labeled com.android.grafika / com.android.grafika.PlayMovieActivity. This is the main UI layer, which is another BufferQueue. Because TextureView renders into the UI layer rather than a separate layer, all of the video-driven updates are displayed here.

Gralloc

The Gralloc allocator HAL hardware/libhardware/include/hardware/gralloc.h performs buffer allocations through usage flags. Usage flags include attributes such as:

  • How often the memory will be accessed from software (CPU)
  • How often the memory will be accessed from hardware (GPU)
  • Whether the memory will be used as an OpenGL ES (GLES) texture
  • Whether the memory will be used by a video encoder

For example, if a producer's buffer format specifies RGBA_8888 pixels, and the producer indicates that the buffer will be accessed from software (meaning an app will touch pixels on the CPU), Gralloc creates a buffer with 4 bytes per pixel in R-G-B-A order. If instead, a producer specifies its buffer will be only accessed from hardware and as a GLES texture, Gralloc can do anything the GLES driver wants, such as BGRA ordering, nonlinear swizzled layouts, and alternative color formats. Allowing the hardware to use its preferred format can improve performance.

Some values can't be combined on certain platforms. For example, the video encoder flag may require YUV pixels, so adding software access and specifying RGBA_8888 fails.

The handle returned by Gralloc can be passed between processes through Binder.

Protected buffers

The Gralloc usage flag GRALLOC_USAGE_PROTECTED allows the graphics buffer to be displayed only through a hardware-protected path. These overlay planes are the only way to display DRM content (DRM-protected buffers can't be accessed by SurfaceFlinger or the OpenGL ES driver).

DRM-protected video can be presented only on an overlay plane. Video players that support protected content must be implemented with SurfaceView. Software running on unprotected hardware can't read or write the buffer; hardware-protected paths must appear on the Hardware Composer overlay (that is, protected videos disappear from the display if Hardware Composer switches to OpenGL ES composition).

For details on protected content, see DRM.