Layers and Displays

Layers and displays are two primitives that represent composition work and interactions with the display hardware.

Layers

A layer is the most important unit of composition. A layer is a combination of a surface and a SurfaceControl. Each layer has a set of properties that define how it interacts with other layers. Layer properties are:

Property Description
Positional Defines where the layer appears on its display. Includes information such as the positions of a layer's edges and its Z order relative to other layers (whether it should be in front of or behind other layers).
Content Defines how content displayed on the layer should be presented within the bounds defined by the positional properties. Includes information such as crop (to expand a portion of the content to fill the bounds of the layer) and transform (to show rotated or flipped content).
Composition Defines how the layer should be composited with other layers. Includes information such as blending mode and a layer-wide alpha value for alpha compositing.
Optimization Provides information not strictly necessary to correctly composite the layer, but that can be used by the Hardware Composer (HWC) device to optimize how it performs composition. Includes information such as the visible region of the layer and which portion of the layer has been updated since the previous frame.

Displays

A display is another important unit of composition. A system can have multiple displays and displays can be added or removed during normal system operations. Displays are added/removed at the request of the HWC or at the request of the framework. The HWC device requests displays be added or removed when an external display is connected or disconnected from the device, which is called hotplugging. Clients request virtual displays, whose contents are rendered into an off-screen buffer instead of to a physical display.

Virtual displays

SurfaceFlinger supports an internal display (built into the phone or tablet), external displays (such as a television connected through HDMI), and one or more virtual displays that make composited output available within the system. Virtual displays can be used to record the screen or send the screen over a network. Frames generated for a virtual display are written to a BufferQueue.

Virtual displays may share the same set of layers as the main display (the layer stack) or have their own set. There's no VSYNC for a virtual display, so the VSYNC for the internal display is used to trigger composition for all displays.

On Hardware Composer (HWC) implementations that support them, virtual displays can be composited with OpenGL ES (GLES), HWC, or both GLES and HWC. On nonsupporting implementations, virtual displays are always composited using GLES.

Case study: screenrecord

The screenrecord command allows the user to record everything that appears on the screen as an .mp4 file on disk. To implement this, the system receives composited frames from SurfaceFlinger, writes them to the video encoder, and then writes the encoded video data to a file. The video codecs are managed by a separate process (mediaserver), so large graphics buffers have to move around the system. To make it more challenging, the goal is to record 60 fps video at full resolution. The key to making this work efficiently is BufferQueue.

The MediaCodec class allows an app to provide data as raw bytes in buffers, or through a surface. When screenrecord requests access to a video encoder, the mediaserver process creates a BufferQueue, connects itself to the consumer side, then passes the producer side back to screenrecord as a surface.

The screenrecord utility then asks SurfaceFlinger to create a virtual display that mirrors the main display (that is, it has all of the same layers), and directs it to send output to the surface that came from the mediaserver process. In this case, SurfaceFlinger is the producer of buffers rather than the consumer.

After the configuration is complete, screenrecord triggers when the encoded data appears. As apps draw, their buffers travel to SurfaceFlinger, which composites them into a single buffer that gets sent directly to the video encoder in the mediaserver process. The full frames are never even seen by the screenrecord process. Internally, the mediaserver process has its own way of moving buffers around that also passes data by handle, minimizing overhead.

Case study: simulate secondary displays

The WindowManager can ask SurfaceFlinger to create a visible layer for which SurfaceFlinger acts as the BufferQueue consumer. It's also possible to ask SurfaceFlinger to create a virtual display, for which SurfaceFlinger acts as the BufferQueue producer.

If you connect a virtual display to a visible layer, a closed loop is created where the composited screen appears in a window. That window is now part of the composited output, so on the next refresh the composited image inside the window shows the window contents as well. To see this in action, enable Developer options in Settings, select Simulate secondary displays, and enable a window. To see secondary displays in action, use screenrecord to capture the act of enabling the display then play it back frame by frame.