Graphics

Android Graphics HAL icon

The Android framework offers a variety of graphics rendering APIs for 2D and 3D that interact with manufacturer implementations of graphics drivers, so it is important to have a good understanding of how those APIs work at a higher level. This page introduces the graphics hardware abstraction layer (HAL) that those drivers are built on. Before continuing with this section, familiarize yourself with the following terms:

canvas (generic term), Canvas (API element)
A canvas is a drawing surface that handles compositing of the actual bits against a bitmap or a Surface object. Canvas has methods for standard computer drawing of bitmaps, lines, circles, rectangles, text, and so on, and is bound to a bitmap or surface. A canvas is the simplest, easiest way to draw 2D objects on the screen. The base class is Canvas.
drawable
A drawable is a compiled visual resource that can be used as a background, title, or other part of the screen. A drawable is typically loaded into another UI element, for example as a background image. A drawable is not able to receive events, but does assign various other properties such as state and scheduling, to enable subclasses such as animation objects or image libraries. Many drawable objects are loaded from drawable resource files — XML or bitmap files that describe the image. Drawable resources are compiled into subclasses of android.graphics.drawable. For more information about drawables and other resources, see Resources.
layout resource
A layout resource is a XML file that describes the layout of an activity screen. For more information, see Layout resource.
nine-patch (9-patch, NinePatch)
A nine-patch is a resizeable bitmap resource that can be used for backgrounds or other images on the device. For more information, see Nine-patch.
OpenGL ES
OpenGL ES is a cross-platform API for rendering 2D and 3D graphics. Android provides OpenGL ES libraries for hardware-accelerated 3D rendering. For 2D rendering, a canvas is the simpler option. OpenGL ES is available in the Android Native Development Kit (NDK). The android.opengl and javax.microedition.khronos.opengles packages expose OpenGL ES functionality.
surface (generic term), Surface (API element)
A surface represents a block of memory that gets composited to the screen. A surface holds a canvas for drawing, and provides various helper methods to draw layers and resize the Surface object. Use the SurfaceView class instead of the Surface class directly.
surface view (generic term), SurfaceView (API element)
A surface view is a View object that wraps a Surface object for drawing, and exposes methods to specify its size and format dynamically. A surface view provides a way to draw independently of the UI thread for resource-intensive operations, such as games or camera previews, but it uses extra memory as a result. A surface view supports both canvas and OpenGL ES graphics. The base class for a SurfaceView object is SurfaceView.
theme
A theme is a set of properties, such as text size and background color, bundled together to define various default display settings. Android provides a few standard themes, listed in R.style and prefaced with Theme_.
view (generic term), View (API element)
A view draws a rectangular area on the screen and handles click, keystroke, and other interaction events. The View class is the base class for most layout components of an activity or dialog screen, such as text boxes and windows. A View object receives calls from its parent object (see ViewGroup) to draw itself, and informs its parent object about its preferred size and located, which might not be respected by the parent. For more information, see View.
view group (generic term), ViewGroup (API element)
A view group groups a set of child views. The view group is responsible for deciding where child views are positioned and how large they can be, as well as for calling each to draw themselves when appropriate. Some view groups are invisible and are for layout only, while others have an intrinsic UI, such as a scrolling list box. View groups are in the widget package, but extend the ViewGroup class.
view hierarchy
A view hieararchy is an arrangement of view and view group objects that defines the user interface for each component of an app. The hierarchy consists of view groups that contain one or more child views or view groups. You can obtain a visual representation of a view hierarchy for debugging and optimization by using the Hierarchy Viewer that is supplied with the Android SDK.
Vulkan
Vulkan is a low-overhead cross-platform API for high-performance 3D graphics.
widget
A widget is one of a set of fully implemented view subclasses that render form elements and other UI components, such as a text box or a popup menu. Because a widget is fully implemented, it handles measuring, drawing itself, and responding to screen events. Widgets are in the android.widget package.
window (generic term), Window (API element)
In an Android app, a window is an object derived from the Window abstract class that specifies the elements of a generic window, such as the look and feel, title bar text, and location and content of menus. Dialogs and activities use an implementation of the Window class to render a Window object. You don't need to implement the Window class or use windows in your app.

App developers draw images to the screen in three ways: with Canvas, OpenGL ES, or Vulkan.

Android graphics components

No matter what rendering API developers use, everything is rendered onto a surface. The surface represents the producer side of a buffer queue that is often consumed by SurfaceFlinger. Every window that is created on the Android platform is backed by a surface. All of the visible surfaces rendered are composited onto the display by SurfaceFlinger.

The following diagram shows how the key components work together:

image rendering components

Figure 1. How surfaces are rendered.

The main components are described below:

Image stream producers

An image stream producer can be anything that produces graphic buffers for consumption. Examples include OpenGL ES, Canvas 2D, and mediaserver video decoders.

Image stream consumers

The most common consumer of image streams is SurfaceFlinger, the system service that consumes the currently visible surfaces and composites them onto the display using information provided by the Window Manager. SurfaceFlinger is the only service that can modify the content of the display. SurfaceFlinger uses OpenGL and the Hardware Composer to compose a group of surfaces.

Other OpenGL ES apps can consume image streams as well, such as the camera app consuming a camera preview image stream. Non-GL apps can be consumers too, for example the ImageReader class.

Hardware Composer

The hardware abstraction for the display subsystem. SurfaceFlinger can delegate certain composition work to the Hardware Composer to offload work from OpenGL and the GPU. SurfaceFlinger acts as just another OpenGL ES client. So when SurfaceFlinger is actively compositing one buffer or two into a third, for instance, it is using OpenGL ES. This makes compositing lower power than having the GPU conduct all computation.

The Hardware Composer HAL conducts the other half of the work and is the central point for all Android graphics rendering. The Hardware Composer must support events, one of which is VSYNC (another is hotplug for plug-and-playHDMI support).

Gralloc

The graphics memory allocator (Gralloc) is needed to allocate memory requested by image producers. For details, see Gralloc HAL.

Data flow

See the following diagram for a depiction of the Android graphics pipeline:

graphics data flow

Figure 2. Graphic data flow through Android

The objects on the left are renderers producing graphics buffers, such as the home screen, status bar, and system UI. SurfaceFlinger is the compositor and Hardware Composer is the composer.

BufferQueue

BufferQueues provide the glue between the Android graphics components. These are a pair of queues that mediate the constant cycle of buffers from the producer to the consumer. Once the producers hand off their buffers, SurfaceFlinger is responsible for compositing everything onto the display.

See the following diagram for the BufferQueue communication process.

BufferQueue communication process

Figure 3. BufferQueue communication process

BufferQueue contains the logic that ties image stream producers and image stream consumers together. Some examples of image producers are the camera previews produced by the camera HAL or OpenGL ES games. Some examples of image consumers are SurfaceFlinger or another app that displays an OpenGL ES stream, such as the camera app displaying the camera viewfinder.

BufferQueue is a data structure that combines a buffer pool with a queue and uses Binder IPC to pass buffers between processes. The producer interface, or what you pass to somebody who wants to generate graphic buffers, is IGraphicBufferProducer (part of SurfaceTexture). BufferQueue is often used to render to a Surface and consume with a GL Consumer, among other tasks.

BufferQueue can operate in three different modes:

Synchronous-like mode - BufferQueue by default operates in a synchronous-like mode, in which every buffer that comes in from the producer goes out at the consumer. No buffer is ever discarded in this mode. And if the producer is too fast and creates buffers faster than they are being drained, it will block and wait for free buffers.

Non-blocking mode - BufferQueue can also operate in a non-blocking mode where it generates an error rather than waiting for a buffer in those cases. No buffer is ever discarded in this mode either. This is useful for avoiding potential deadlocks in application software that may not understand the complex dependencies of the graphics framework.

Discard mode - Finally, BufferQueue may be configured to discard old buffers rather than generate errors or wait. For instance, if conducting GL rendering to a texture view and drawing as quickly as possible, buffers must be dropped.

To conduct most of this work, SurfaceFlinger acts as just another OpenGL ES client. So when SurfaceFlinger is actively compositing one buffer or two into a third, for instance, it is using OpenGL ES.

The Hardware Composer HAL conducts the other half of the work. This HAL acts as the central point for all Android graphics rendering.