| <html devsite> |
| <head> |
| <title>SurfaceView and GLSurfaceView</title> |
| <meta name="project_path" value="/_project.yaml" /> |
| <meta name="book_path" value="/_book.yaml" /> |
| </head> |
| <body> |
| <!-- |
| Copyright 2017 The Android Open Source Project |
| |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| |
| |
| |
| <p>The Android app framework UI is based on a hierarchy of objects that start |
| with View. All UI elements go through a complicated measurement and layout |
| process that fits them into a rectangular area, and all visible View objects are |
| rendered to a SurfaceFlinger-created Surface that was set up by the |
| WindowManager when the app was brought to the foreground. The app's UI thread |
| performs layout and rendering to a single buffer (regardless of the number of |
| Layouts and Views and whether or not the Views are hardware-accelerated).</p> |
| |
| <p>A SurfaceView takes the same parameters as other views, so you can give it a |
| position and size, and fit other elements around it. When it comes time to |
| render, however, the contents are completely transparent; The View part of a |
| SurfaceView is just a see-through placeholder.</p> |
| |
| <p>When the SurfaceView's View component is about to become visible, the |
| framework asks the WindowManager to ask SurfaceFlinger to create a new Surface. |
| (This doesn't happen synchronously, which is why you should provide a callback |
| that notifies you when the Surface creation finishes.) By default, the new |
| Surface is placed behind the app UI Surface, but the default Z-ordering can be |
| overridden to put the Surface on top.</p> |
| |
| <p>Whatever you render onto this Surface will be composited by SurfaceFlinger, |
| not by the app. This is the real power of SurfaceView: The Surface you get can |
| be rendered by a separate thread or a separate process, isolated from any |
| rendering performed by the app UI, and the buffers go directly to |
| SurfaceFlinger. You can't totally ignore the UI thread—you still have to |
| coordinate with the Activity lifecycle and you may need to adjust something if |
| the size or position of the View changes—but you have a whole Surface all |
| to yourself. Blending with the app UI and other layers is handled by the |
| Hardware Composer.</p> |
| |
| <p>The new Surface is the producer side of a BufferQueue, whose consumer is a |
| SurfaceFlinger layer. You can update the Surface with any mechanism that can |
| feed a BufferQueue, such as surface-supplied Canvas functions, attach an |
| EGLSurface and draw on it with GLES, or configure a MediaCodec video decoder to |
| write to it.</p> |
| |
| <h2 id=composition>Composition and the Hardware Scaler</h2> |
| |
| <p>Let's take a closer look at <code>dumpsys SurfaceFlinger</code>. The |
| following output was taken while playing a movie in Grafika's "Play video |
| (SurfaceView)" activity on a Nexus 5 in portrait orientation; the video is QVGA |
| (320x240):</p> |
| <p><pre> |
| type | source crop | frame name |
| ------------+-----------------------------------+-------------------------------- |
| HWC | [ 0.0, 0.0, 320.0, 240.0] | [ 48, 411, 1032, 1149] SurfaceView |
| HWC | [ 0.0, 75.0, 1080.0, 1776.0] | [ 0, 75, 1080, 1776] com.android.grafika/com.android.grafika.PlayMovieSurfaceActivity |
| HWC | [ 0.0, 0.0, 1080.0, 75.0] | [ 0, 0, 1080, 75] StatusBar |
| HWC | [ 0.0, 0.0, 1080.0, 144.0] | [ 0, 1776, 1080, 1920] NavigationBar |
| FB TARGET | [ 0.0, 0.0, 1080.0, 1920.0] | [ 0, 0, 1080, 1920] HWC_FRAMEBUFFER_TARGET |
| </pre></p> |
| |
| <ul> |
| <li>The <strong>list order</strong> is back to front: the SurfaceView's Surface |
| is in the back, the app UI layer sits on top of that, followed by the status and |
| navigation bars that are above everything else.</li> |
| <li>The <strong>source crop</strong> values indicate the portion of the |
| Surface's buffer that SurfaceFlinger will display. The app UI was given a |
| Surface equal to the full size of the display (1080x1920), but as there is no |
| point rendering and compositing pixels that will be obscured by the status and |
| navigation bars, the source is cropped to a rectangle that starts 75 pixels from |
| the top and ends 144 pixels from the bottom. The status and navigation bars have |
| smaller Surfaces, and the source crop describes a rectangle that begins at the |
| top left (0,0) and spans their content.</li> |
| <li>The <strong>frame</strong> values specify the rectangle where pixels |
| appear on the display. For the app UI layer, the frame matches the source crop |
| because we are copying (or overlaying) a portion of a display-sized layer to the |
| same location in another display-sized layer. For the status and navigation |
| bars, the size of the frame rectangle is the same, but the position is adjusted |
| so the navigation bar appears at the bottom of the screen.</li> |
| <li>The <strong>SurfaceView layer</strong> holds our video content. The source crop |
| matches the video size, which SurfaceFlinger knows because the MediaCodec |
| decoder (the buffer producer) is dequeuing buffers that size. The frame |
| rectangle has a completely different size—984x738.</li> |
| </ul> |
| |
| <p>SurfaceFlinger handles size differences by scaling the buffer contents to |
| fill the frame rectangle, upscaling or downscaling as needed. This particular |
| size was chosen because it has the same aspect ratio as the video (4:3), and is |
| as wide as possible given the constraints of the View layout (which includes |
| some padding at the edges of the screen for aesthetic reasons).</p> |
| |
| <p>If you started playing a different video on the same Surface, the underlying |
| BufferQueue would reallocate buffers to the new size automatically, and |
| SurfaceFlinger would adjust the source crop. If the aspect ratio of the new |
| video is different, the app would need to force a re-layout of the View to match |
| it, which causes the WindowManager to tell SurfaceFlinger to update the frame |
| rectangle.</p> |
| |
| <p>If you're rendering on the Surface through some other means (such as GLES), |
| you can set the Surface size using the <code>SurfaceHolder#setFixedSize()</code> |
| call. For example, you could configure a game to always render at 1280x720, |
| which would significantly reduce the number of pixels that must be touched to |
| fill the screen on a 2560x1440 tablet or 4K television. The display processor |
| handles the scaling. If you don't want to letter- or pillar-box your game, you |
| could adjust the game's aspect ratio by setting the size so that the narrow |
| dimension is 720 pixels but the long dimension is set to maintain the aspect |
| ratio of the physical display (e.g. 1152x720 to match a 2560x1600 display). |
| For an example of this approach, see Grafika's "Hardware scaler exerciser" |
| activity.</p> |
| |
| <h2 id=glsurfaceview>GLSurfaceView</h2> |
| |
| <p>The GLSurfaceView class provides helper classes for managing EGL contexts, |
| inter-thread communication, and interaction with the Activity lifecycle. That's |
| it. You do not need to use a GLSurfaceView to use GLES.</p> |
| |
| <p>For example, GLSurfaceView creates a thread for rendering and configures an |
| EGL context there. The state is cleaned up automatically when the activity |
| pauses. Most apps won't need to know anything about EGL to use GLES with |
| GLSurfaceView.</p> |
| |
| <p>In most cases, GLSurfaceView is very helpful and can make working with GLES |
| easier. In some situations, it can get in the way. Use it if it helps, don't |
| if it doesn't.</p> |
| |
| <h2 id=activity>SurfaceView and the Activity Lifecycle</h2> |
| |
| <p>When using a SurfaceView, it's considered good practice to render the Surface |
| from a thread other than the main UI thread. This raises some questions about |
| the interaction between that thread and the Activity lifecycle.</p> |
| |
| <p>For an Activity with a SurfaceView, there are two separate but interdependent |
| state machines:</p> |
| |
| <ol> |
| <li>Application onCreate/onResume/onPause</li> |
| <li>Surface created/changed/destroyed</li> |
| </ol> |
| |
| <p>When the Activity starts, you get callbacks in this order:</p> |
| |
| <ul> |
| <li>onCreate</li> |
| <li>onResume</li> |
| <li>surfaceCreated</li> |
| <li>surfaceChanged</li> |
| </ul> |
| |
| <p>If you hit back you get:</p> |
| |
| <ul> |
| <li>onPause</li> |
| <li>surfaceDestroyed (called just before the Surface goes away)</li> |
| </ul> |
| |
| <p>If you rotate the screen, the Activity is torn down and recreated and you |
| get the full cycle. You can tell it's a quick restart by checking |
| <code>isFinishing()</code>. It might be possible to start/stop an Activity so |
| quickly that <code>surfaceCreated()</code> might actually happen after |
| <code>onPause()</code>.</p> |
| |
| <p>If you tap the power button to blank the screen, you get only |
| <code>onPause()</code>—no <code>surfaceDestroyed()</code>. The Surface |
| remains alive, and rendering can continue. You can even keep getting |
| Choreographer events if you continue to request them. If you have a lock |
| screen that forces a different orientation, your Activity may be restarted when |
| the device is unblanked; but if not, you can come out of screen-blank with the |
| same Surface you had before.</p> |
| |
| <p>This raises a fundamental question when using a separate renderer thread with |
| SurfaceView: Should the lifespan of the thread be tied to that of the Surface or |
| the Activity? The answer depends on what you want to happen when the screen |
| goes blank: (1) start/stop the thread on Activity start/stop or (2) start/stop |
| the thread on Surface create/destroy.</p> |
| |
| <p>Option 1 interacts well with the app lifecycle. We start the renderer thread |
| in <code>onResume()</code> and stop it in <code>onPause()</code>. It gets a bit |
| awkward when creating and configuring the thread because sometimes the Surface |
| will already exist and sometimes it won't (e.g. it's still alive after toggling |
| the screen with the power button). We have to wait for the surface to be |
| created before we do some initialization in the thread, but we can't simply do |
| it in the <code>surfaceCreated()</code> callback because that won't fire again |
| if the Surface didn't get recreated. So we need to query or cache the Surface |
| state, and forward it to the renderer thread.</p> |
| |
| <p class="note"><strong>Note:</strong> Be careful when passing objects |
| between threads. It is best to pass the Surface or SurfaceHolder through a |
| Handler message (rather than just stuffing it into the thread) to avoid issues |
| on multi-core systems. For details, refer to |
| <a href="http://developer.android.com/training/articles/smp.html">Android |
| SMP Primer</a>.</p> |
| |
| <p>Option 2 is appealing because the Surface and the renderer are logically |
| intertwined. We start the thread after the Surface has been created, which |
| avoids some inter-thread communication concerns, and Surface created/changed |
| messages are simply forwarded. We need to ensure rendering stops when the |
| screen goes blank and resumes when it un-blanks; this could be a simple matter |
| of telling Choreographer to stop invoking the frame draw callback. Our |
| <code>onResume()</code> will need to resume the callbacks if and only if the |
| renderer thread is running. It may not be so trivial though—if we animate |
| based on elapsed time between frames, we could have a very large gap when the |
| next event arrives; an explicit pause/resume message may be desirable.</p> |
| |
| <p class="note"><strong>Note:</strong> For an example of Option 2, see Grafika's |
| "Hardware scaler exerciser."</p> |
| |
| <p>Both options are primarily concerned with how the renderer thread is |
| configured and whether it's executing. A related concern is extracting state |
| from the thread when the Activity is killed (in <code>onPause()</code> or |
| <code>onSaveInstanceState()</code>); in such cases, Option 1 works best because |
| after the renderer thread has been joined its state can be accessed without |
| synchronization primitives.</p> |
| |
| </body> |
| </html> |