| <html devsite> |
| <head> |
| <title>HAL Subsystem</title> |
| <meta name="project_path" value="/_project.yaml" /> |
| <meta name="book_path" value="/_book.yaml" /> |
| </head> |
| <body> |
| <!-- |
| Copyright 2017 The Android Open Source Project |
| |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| |
| |
| |
| <h2 id="requests">Requests</h2> |
| <p> The app framework issues requests for captured results to the camera subsystem. |
| One request corresponds to one set of results. A request encapsulates all |
| configuration information about the capturing and processing of those results. |
| This includes things such as resolution and pixel format; manual sensor, lens, |
| and flash control; 3A operating modes; RAW to YUV processing control; and |
| statistics generation. This allows for much more control over the results' |
| output and processing. Multiple requests can be in flight at once, and |
| submitting requests is non-blocking. And the requests are always processed in |
| the order they are received.</p> |
| <img src="images/camera_model.png" alt="Camera request model" id="figure1" /> |
| <p class="img-caption"> |
| <strong>Figure 1.</strong> Camera model |
| </p> |
| <h2 id="hal-subsystem">The HAL and camera subsystem</h2> |
| <p> The camera subsystem includes the implementations for components in the camera |
| pipeline such as the 3A algorithm and processing controls. The camera HAL |
| provides interfaces for you to implement your versions of these components. To |
| maintain cross-platform compatibility between multiple device manufacturers and |
| Image Signal Processor (ISP, or camera sensor) vendors, the camera pipeline |
| model is virtual and does not directly correspond to any real ISP. However, it |
| is similar enough to real processing pipelines so that you can map it to your |
| hardware efficiently. In addition, it is abstract enough to allow for multiple |
| different algorithms and orders of operation without compromising either |
| quality, efficiency, or cross-device compatibility.</p> |
| <p>The camera pipeline also supports triggers that the app framework can initiate |
| to turn on things such as auto-focus. It also sends notifications back to the |
| app framework, notifying apps of events such as an auto-focus lock or errors.</p> |
| <img src="images/camera_hal.png" alt="Camera hardware abstraction layer" id="figure2" /> |
| <p class="img-caption"> |
| <strong>Figure 2.</strong> Camera pipeline</p> |
| <p>Please note, some image processing blocks shown in the diagram above are not |
| well-defined in the initial release. The camera pipeline makes the following |
| assumptions:</p> |
| <ul> |
| <li>RAW Bayer output undergoes no processing inside the ISP.</li> |
| <li>Statistics are generated based off the raw sensor data.</li> |
| <li>The various processing blocks that convert raw sensor data to YUV are in an |
| arbitrary order.</li> |
| <li>While multiple scale and crop units are shown, all scaler units share the |
| output region controls (digital zoom). However, each unit may have a different |
| output resolution and pixel format.</li> |
| </ul> |
| <p><strong>Summary of API use</strong><br/> |
| This is a brief summary of the steps for using the Android camera API. See the |
| Startup and expected operation sequence section for a detailed breakdown of |
| these steps, including API calls.</p> |
| <ol> |
| <li>Listen for and enumerate camera devices.</li> |
| <li>Open device and connect listeners.</li> |
| <li>Configure outputs for target use case (such as still capture, recording, |
| etc.).</li> |
| <li>Create request(s) for target use case.</li> |
| <li>Capture/repeat requests and bursts.</li> |
| <li>Receive result metadata and image data.</li> |
| <li>When switching use cases, return to step 3.</li> |
| </ol> |
| <p><strong>HAL operation summary</strong></p> |
| <ul> |
| <li>Asynchronous requests for captures come from the framework.</li> |
| <li>HAL device must process requests in order. And for each request, produce |
| output result metadata, and one or more output image buffers.</li> |
| <li>First-in, first-out for requests and results, and for streams referenced by |
| subsequent requests. </li> |
| <li>Timestamps must be identical for all outputs from a given request, so that the |
| framework can match them together if needed. </li> |
| <li>All capture configuration and state (except for the 3A routines) is |
| encapsulated in the requests and results.</li> |
| </ul> |
| <img src="images/camera-hal-overview.png" alt="Camera HAL overview" id="figure3" /> |
| <p class="img-caption"> |
| <strong>Figure 3.</strong> Camera HAL overview |
| </p> |
| <h2 id="startup">Startup and expected operation sequence</h2> |
| <p>This section contains a detailed explanation of the steps expected when using |
| the camera API. Please see <a href="https://android.googlesource.com/platform/hardware/interfaces/+/master/camera/">platform/hardware/interfaces/camera/</a> for HIDL interface |
| definitions.</p> |
| |
| <h3 id="open-camera-device">Enumerating, opening camera devices and |
| creating an active session</h3> |
| <ol> |
| <li>After initialization, the framework starts listening for any present |
| camera providers that implement the |
| <code><a href="https://android.googlesource.com/platform/hardware/interfaces/+/master/camera/provider/2.4/ICameraProvider.hal">ICameraProvider</a></code> interface. If such provider or |
| providers are present, the framework will try to establish a connection.</li> |
| <li>The framework enumerates the camera devices via |
| <code>ICameraProvider::getCameraIdList()</code>.</li> |
| <li>The framework instantiates a new <code>ICameraDevice</code> by calling the respective |
| <code>ICameraProvider::getCameraDeviceInterface_VX_X()</code>.</li> |
| <li>The framework calls <code>ICameraDevice::open()</code> to create a new |
| active capture session ICameraDeviceSession.</li> |
| </ol> |
| |
| <h3 id="use-active-session">Using an active camera session</h3> |
| |
| <ol> |
| <li>The framework calls <code>ICameraDeviceSession::configureStreams()</code> |
| with a list of input/output streams to the HAL device.</li> |
| <li>The framework requests default settings for some use cases with |
| calls to <code>ICameraDeviceSession::constructDefaultRequestSettings()</code>. |
| This may occur at any time after the <code>ICameraDeviceSession</code> is |
| created by <code>ICameraDevice::open</code>. |
| </li> |
| <li>The framework constructs and sends the first capture request to the HAL with |
| settings based on one of the sets of default settings, and with at least one |
| output stream that has been registered earlier by the framework. This is sent |
| to the HAL with <code>ICameraDeviceSession::processCaptureRequest()</code>. |
| The HAL must block the return of this call until it is ready for the next |
| request to be sent.</li> |
| <li>The framework continues to submit requests and calls |
| <code>ICameraDeviceSession::constructDefaultRequestSettings()</code> to get |
| default settings buffers for other use cases as necessary.</li> |
| <li>When the capture of a request begins (sensor starts exposing for the |
| capture), the HAL calls <code>ICameraDeviceCallback::notify()</code> with |
| the SHUTTER message, including the frame number and the timestamp for start |
| of exposure. This notify callback does not have to happen before the first |
| <code>processCaptureResult()</code> call for a request, but no results are |
| delivered to an application for a capture until after |
| <code>notify()</code> for that capture is called. |
| </li> |
| <li>After some pipeline delay, the HAL begins to return completed captures to |
| the framework with <code>ICameraDeviceCallback::processCaptureResult()</code>. |
| These are returned in the same order as the requests were submitted. Multiple |
| requests can be in flight at once, depending on the pipeline depth of the |
| camera HAL device.</li> |
| </ol> |
| |
| <p>After some time, one of the following will occur:</p> |
| <ul> |
| <li>The framework may stop submitting new requests, wait for |
| the existing captures to complete (all buffers filled, all results |
| returned), and then call <code>ICameraDeviceSession::configureStreams()</code> |
| again. This resets the camera hardware and pipeline for a new set of |
| input/output streams. Some streams may be reused from the previous |
| configuration. The framework then continues from the first capture request |
| to the HAL, if at least one |
| registered output stream remains. (Otherwise, |
| <code>ICameraDeviceSession::configureStreams()</code> is required first.)</li> |
| <li>The framework may call <code>ICameraDeviceSession::close()</code> |
| to end the camera session. This may be called at any time when no other calls |
| from the framework are active, although the call may block until all |
| in-flight captures have completed (all results returned, all buffers |
| filled). After the <code>close()</code> call returns, no more calls to |
| <code>ICameraDeviceCallback</code> are allowed from the HAL. Once the |
| <code>close()</code> call is underway, the framework may not call any other |
| HAL device functions.</li> |
| <li>In case of an error or other asynchronous event, the HAL must call |
| <code>ICameraDeviceCallback::notify()</code> with the appropriate |
| error/event message. |
| After returning from a fatal device-wide error notification, the HAL should |
| act as if <code>close()</code> had been called on it. However, the HAL must |
| either cancel |
| or complete all outstanding captures before calling <code>notify()</code>, |
| so that once |
| <code>notify()</code> is called with a fatal error, the framework will not |
| receive further callbacks from the device. Methods besides |
| <code>close()</code> should return |
| -ENODEV or NULL after the <code>notify()</code> method returns from a fatal |
| error message.</li> |
| </ul> |
| <img src="images/camera-ops-flow.png" alt="Camera operations flow" id="figure4" width="485"/> |
| <p class="img-caption"> |
| <strong>Figure 4.</strong> Camera operational flow |
| </p> |
| <h2 id="hardware-levels">Hardware levels</h2> |
| <p>Camera devices can implement several hardware levels depending on their |
| capabilities. For more information, see |
| <a href="https://developer.android.com/reference/android/hardware/camera2/CameraCharacteristics#INFO_SUPPORTED_HARDWARE_LEVEL">supported hardware level</a>.</p> |
| <h2 id="interaction">Interaction between the application capture request, 3A |
| control, and the processing pipeline</h2> |
| <p>Depending on the settings in the 3A control block, the camera pipeline ignores |
| some of the parameters in the application's capture request and uses the values |
| provided by the 3A control routines instead. For example, when auto-exposure is |
| active, the exposure time, frame duration, and sensitivity parameters of the |
| sensor are controlled by the platform 3A algorithm, and any app-specified values |
| are ignored. The values chosen for the frame by the 3A routines must be reported |
| in the output metadata. The following table describes the different modes of the |
| 3A control block and the properties that are controlled by these modes. See |
| the <a href="https://android.googlesource.com/platform/system/media/+/master/camera/docs/docs.html">platform/system/media/camera/docs/docs.html</a> file for definitions of these properties.</p> |
| <table> |
| <tr> |
| <th>Parameter</th> |
| <th>State</th> |
| <th>Properties controlled</th> |
| </tr> |
| <tr> |
| <td>android.control.aeMode</td> |
| <td>OFF</td> |
| <td>None</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>ON</td> |
| <td>android.sensor.exposureTime |
| android.sensor.frameDuration |
| android.sensor.sensitivity |
| android.lens.aperture (if supported) |
| android.lens.filterDensity (if supported)</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>ON_AUTO_FLASH</td> |
| <td>Everything is ON, plus android.flash.firingPower, android.flash.firingTime, and android.flash.mode</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>ON_ALWAYS_FLASH</td> |
| <td>Same as ON_AUTO_FLASH</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>ON_AUTO_FLASH_RED_EYE</td> |
| <td>Same as ON_AUTO_FLASH</td> |
| </tr> |
| <tr> |
| <td>android.control.awbMode</td> |
| <td>OFF</td> |
| <td>None</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>WHITE_BALANCE_*</td> |
| <td>android.colorCorrection.transform. Platform-specific adjustments if android.colorCorrection.mode is FAST or HIGH_QUALITY.</td> |
| </tr> |
| <tr> |
| <td>android.control.afMode</td> |
| <td>OFF</td> |
| <td>None</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>FOCUS_MODE_*</td> |
| <td>android.lens.focusDistance</td> |
| </tr> |
| <tr> |
| <td>android.control.videoStabilization</td> |
| <td>OFF</td> |
| <td>None</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>ON</td> |
| <td>Can adjust android.scaler.cropRegion to implement video stabilization</td> |
| </tr> |
| <tr> |
| <td>android.control.mode</td> |
| <td>OFF</td> |
| <td>AE, AWB, and AF are disabled</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>AUTO</td> |
| <td>Individual AE, AWB, and AF settings are used</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>SCENE_MODE_*</td> |
| <td>Can override all parameters listed above. Individual 3A controls are disabled.</td> |
| </tr> |
| </table> |
| <p>The controls in the Image Processing block in Figure 2 all operate on a |
| similar principle, and generally each block has three modes:</p> |
| <ul> |
| <li>OFF: This processing block is disabled. The demosaic, color correction, and |
| tone curve adjustment blocks cannot be disabled.</li> |
| <li>FAST: In this mode, the processing block may not slow down the output frame |
| rate compared to OFF mode, but should otherwise produce the best-quality |
| output it can given that restriction. Typically, this would be used for |
| preview or video recording modes, or burst capture for still images. On some |
| devices, this may be equivalent to OFF mode (no processing can be done without |
| slowing down the frame rate), and on some devices, this may be equivalent to |
| HIGH_QUALITY mode (best quality still does not slow down frame rate).</li> |
| <li>HIGH_QUALITY: In this mode, the processing block should produce the best |
| quality result possible, slowing down the output frame rate as needed. |
| Typically, this would be used for high-quality still capture. Some blocks |
| include a manual control which can be optionally selected instead of FAST or |
| HIGH_QUALITY. For example, the color correction block supports a color |
| transform matrix, while the tone curve adjustment supports an arbitrary global |
| tone mapping curve.</li> |
| </ul> |
| <p>The maximum frame rate that can be supported by a camera subsystem is a function |
| of many factors:</p> |
| <ul> |
| <li>Requested resolutions of output image streams</li> |
| <li>Availability of binning/skipping modes on the imager</li> |
| <li>The bandwidth of the imager interface</li> |
| <li>The bandwidth of the various ISP processing blocks</li> |
| </ul> |
| <p>Since these factors can vary greatly between different ISPs and sensors, the |
| camera HAL interface tries to abstract the bandwidth restrictions into as simple |
| model as possible. The model presented has the following characteristics:</p> |
| <ul> |
| <li>The image sensor is always configured to output the smallest resolution |
| possible given the application's requested output stream sizes. The smallest |
| resolution is defined as being at least as large as the largest requested |
| output stream size.</li> |
| <li>Since any request may use any or all the currently configured output streams, |
| the sensor and ISP must be configured to support scaling a single capture to |
| all the streams at the same time. </li> |
| <li>JPEG streams act like processed YUV streams for requests for which they are |
| not included; in requests in which they are directly referenced, they act as |
| JPEG streams.</li> |
| <li>The JPEG processor can run concurrently to the rest of the camera pipeline but |
| cannot process more than one capture at a time.</li> |
| </ul> |
| |
| </body> |
| </html> |