The Android app framework UI is based on a hierarchy of objects that start with View. All UI elements go through a complicated measurement and layout process that fits them into a rectangular area, and all visible View objects are rendered to a SurfaceFlinger-created Surface that was set up by the WindowManager when the app was brought to the foreground. The app's UI thread performs layout and rendering to a single buffer (regardless of the number of Layouts and Views and whether or not the Views are hardware-accelerated).
A SurfaceView takes the same parameters as other views, so you can give it a position and size, and fit other elements around it. When it comes time to render, however, the contents are completely transparent; The View part of a SurfaceView is just a see-through placeholder.
When the SurfaceView's View component is about to become visible, the framework asks the WindowManager to ask SurfaceFlinger to create a new Surface. (This doesn't happen synchronously, which is why you should provide a callback that notifies you when the Surface creation finishes.) By default, the new Surface is placed behind the app UI Surface, but the default Z-ordering can be overridden to put the Surface on top.
Whatever you render onto this Surface will be composited by SurfaceFlinger, not by the app. This is the real power of SurfaceView: The Surface you get can be rendered by a separate thread or a separate process, isolated from any rendering performed by the app UI, and the buffers go directly to SurfaceFlinger. You can't totally ignore the UI thread—you still have to coordinate with the Activity lifecycle and you may need to adjust something if the size or position of the View changes—but you have a whole Surface all to yourself. Blending with the app UI and other layers is handled by the Hardware Composer.
The new Surface is the producer side of a BufferQueue, whose consumer is a SurfaceFlinger layer. You can update the Surface with any mechanism that can feed a BufferQueue, such as surface-supplied Canvas functions, attach an EGLSurface and draw on it with GLES, or configure a MediaCodec video decoder to write to it.
Composition and the Hardware Scaler
Let's take a closer look at
dumpsys SurfaceFlinger. The
following output was taken while playing a movie in Grafika's "Play video
(SurfaceView)" activity on a Nexus 5 in portrait orientation; the video is QVGA
type | source crop | frame name ------------+-----------------------------------+-------------------------------- HWC | [ 0.0, 0.0, 320.0, 240.0] | [ 48, 411, 1032, 1149] SurfaceView HWC | [ 0.0, 75.0, 1080.0, 1776.0] | [ 0, 75, 1080, 1776] com.android.grafika/com.android.grafika.PlayMovieSurfaceActivity HWC | [ 0.0, 0.0, 1080.0, 75.0] | [ 0, 0, 1080, 75] StatusBar HWC | [ 0.0, 0.0, 1080.0, 144.0] | [ 0, 1776, 1080, 1920] NavigationBar FB TARGET | [ 0.0, 0.0, 1080.0, 1920.0] | [ 0, 0, 1080, 1920] HWC_FRAMEBUFFER_TARGET
- The list order is back to front: the SurfaceView's Surface is in the back, the app UI layer sits on top of that, followed by the status and navigation bars that are above everything else.
- The source crop values indicate the portion of the Surface's buffer that SurfaceFlinger will display. The app UI was given a Surface equal to the full size of the display (1080x1920), but as there is no point rendering and compositing pixels that will be obscured by the status and navigation bars, the source is cropped to a rectangle that starts 75 pixels from the top and ends 144 pixels from the bottom. The status and navigation bars have smaller Surfaces, and the source crop describes a rectangle that begins at the top left (0,0) and spans their content.
- The frame values specify the rectangle where pixels appear on the display. For the app UI layer, the frame matches the source crop because we are copying (or overlaying) a portion of a display-sized layer to the same location in another display-sized layer. For the status and navigation bars, the size of the frame rectangle is the same, but the position is adjusted so the navigation bar appears at the bottom of the screen.
- The SurfaceView layer holds our video content. The source crop matches the video size, which SurfaceFlinger knows because the MediaCodec decoder (the buffer producer) is dequeuing buffers that size. The frame rectangle has a completely different size—984x738.
SurfaceFlinger handles size differences by scaling the buffer contents to fill the frame rectangle, upscaling or downscaling as needed. This particular size was chosen because it has the same aspect ratio as the video (4:3), and is as wide as possible given the constraints of the View layout (which includes some padding at the edges of the screen for aesthetic reasons).
If you started playing a different video on the same Surface, the underlying BufferQueue would reallocate buffers to the new size automatically, and SurfaceFlinger would adjust the source crop. If the aspect ratio of the new video is different, the app would need to force a re-layout of the View to match it, which causes the WindowManager to tell SurfaceFlinger to update the frame rectangle.
If you're rendering on the Surface through some other means (such as GLES),
you can set the Surface size using the
call. For example, you could configure a game to always render at 1280x720,
which would significantly reduce the number of pixels that must be touched to
fill the screen on a 2560x1440 tablet or 4K television. The display processor
handles the scaling. If you don't want to letter- or pillar-box your game, you
could adjust the game's aspect ratio by setting the size so that the narrow
dimension is 720 pixels but the long dimension is set to maintain the aspect
ratio of the physical display (e.g. 1152x720 to match a 2560x1600 display).
For an example of this approach, see Grafika's "Hardware scaler exerciser"
The GLSurfaceView class provides helper classes for managing EGL contexts, inter-thread communication, and interaction with the Activity lifecycle. That's it. You do not need to use a GLSurfaceView to use GLES.
For example, GLSurfaceView creates a thread for rendering and configures an EGL context there. The state is cleaned up automatically when the activity pauses. Most apps won't need to know anything about EGL to use GLES with GLSurfaceView.
In most cases, GLSurfaceView is very helpful and can make working with GLES easier. In some situations, it can get in the way. Use it if it helps, don't if it doesn't.
SurfaceView and the Activity Lifecycle
When using a SurfaceView, it's considered good practice to render the Surface from a thread other than the main UI thread. This raises some questions about the interaction between that thread and the Activity lifecycle.
For an Activity with a SurfaceView, there are two separate but interdependent state machines:
- Application onCreate/onResume/onPause
- Surface created/changed/destroyed
When the Activity starts, you get callbacks in this order:
If you hit back you get:
- surfaceDestroyed (called just before the Surface goes away)
If you rotate the screen, the Activity is torn down and recreated and you
get the full cycle. You can tell it's a quick restart by checking
isFinishing(). It might be possible to start/stop an Activity so
surfaceCreated() might actually happen after
If you tap the power button to blank the screen, you get only
surfaceDestroyed(). The Surface
remains alive, and rendering can continue. You can even keep getting
Choreographer events if you continue to request them. If you have a lock
screen that forces a different orientation, your Activity may be restarted when
the device is unblanked; but if not, you can come out of screen-blank with the
same Surface you had before.
This raises a fundamental question when using a separate renderer thread with SurfaceView: Should the lifespan of the thread be tied to that of the Surface or the Activity? The answer depends on what you want to happen when the screen goes blank: (1) start/stop the thread on Activity start/stop or (2) start/stop the thread on Surface create/destroy.
Option 1 interacts well with the app lifecycle. We start the renderer thread
onResume() and stop it in
onPause(). It gets a bit
awkward when creating and configuring the thread because sometimes the Surface
will already exist and sometimes it won't (e.g. it's still alive after toggling
the screen with the power button). We have to wait for the surface to be
created before we do some initialization in the thread, but we can't simply do
it in the
surfaceCreated() callback because that won't fire again
if the Surface didn't get recreated. So we need to query or cache the Surface
state, and forward it to the renderer thread.
Note: Be careful when passing objects between threads. It is best to pass the Surface or SurfaceHolder through a Handler message (rather than just stuffing it into the thread) to avoid issues on multi-core systems. For details, refer to Android SMP Primer.
Option 2 is appealing because the Surface and the renderer are logically
intertwined. We start the thread after the Surface has been created, which
avoids some inter-thread communication concerns, and Surface created/changed
messages are simply forwarded. We need to ensure rendering stops when the
screen goes blank and resumes when it un-blanks; this could be a simple matter
of telling Choreographer to stop invoking the frame draw callback. Our
onResume() will need to resume the callbacks if and only if the
renderer thread is running. It may not be so trivial though—if we animate
based on elapsed time between frames, we could have a very large gap when the
next event arrives; an explicit pause/resume message may be desirable.
Note: For an example of Option 2, see Grafika's "Hardware scaler exerciser."
Both options are primarily concerned with how the renderer thread is
configured and whether it's executing. A related concern is extracting state
from the thread when the Activity is killed (in
onSaveInstanceState()); in such cases, Option 1 works best because
after the renderer thread has been joined its state can be accessed without