| page.title=HAL subsystem |
| @jd:body |
| |
| <!-- |
| Copyright 2013 The Android Open Source Project |
| |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <div id="qv-wrapper"> |
| <div id="qv"> |
| <h2>In this document</h2> |
| <ol id="auto-toc"> |
| </ol> |
| </div> |
| </div> |
| |
| <h2 id="requests">Requests</h2> |
| <p> The app framework issues requests for captured results to the camera subsystem. |
| One request corresponds to one set of results. A request encapsulates all |
| configuration information about the capturing and processing of those results. |
| This includes things such as resolution and pixel format; manual sensor, lens, |
| and flash control; 3A operating modes; RAW to YUV processing control; and |
| statistics generation. This allows for much more control over the results' |
| output and processing. Multiple requests can be in flight at once, and |
| submitting requests is non-blocking. And the requests are always processed in |
| the order they are received.<br/> |
| <img src="images/camera_model.png" alt="Camera request model" id="figure1" /> |
| <p class="img-caption"> |
| <strong>Figure 1.</strong> Camera model |
| </p> |
| <h2 id="hal-subsystem">The HAL and camera subsystem</h2> |
| <p> The camera subsystem includes the implementations for components in the camera |
| pipeline such as the 3A algorithm and processing controls. The camera HAL |
| provides interfaces for you to implement your versions of these components. To |
| maintain cross-platform compatibility between multiple device manufacturers and |
| Image Signal Processor (ISP, or camera sensor) vendors, the camera pipeline |
| model is virtual and does not directly correspond to any real ISP. However, it |
| is similar enough to real processing pipelines so that you can map it to your |
| hardware efficiently. In addition, it is abstract enough to allow for multiple |
| different algorithms and orders of operation without compromising either |
| quality, efficiency, or cross-device compatibility.<br/> |
| The camera pipeline also supports triggers that the app framework can initiate |
| to turn on things such as auto-focus. It also sends notifications back to the |
| app framework, notifying apps of events such as an auto-focus lock or errors.<br/> |
| <img src="images/camera_hal.png" alt="Camera hardware abstraction layer" id="figure2" /> |
| <p class="img-caption"> |
| <strong>Figure 2.</strong> Camera pipeline |
| </p> |
| Please note, some image processing blocks shown in the diagram above are not |
| well-defined in the initial release.<br/> |
| The camera pipeline makes the following assumptions:</p> |
| <ul> |
| <li>RAW Bayer output undergoes no processing inside the ISP.</li> |
| <li>Statistics are generated based off the raw sensor data.</li> |
| <li>The various processing blocks that convert raw sensor data to YUV are in an |
| arbitrary order.</li> |
| <li>While multiple scale and crop units are shown, all scaler units share the |
| output region controls (digital zoom). However, each unit may have a different |
| output resolution and pixel format.</li> |
| </ul> |
| <p><strong>Summary of API use</strong><br/> |
| This is a brief summary of the steps for using the Android camera API. See the |
| Startup and expected operation sequence section for a detailed breakdown of |
| these steps, including API calls.</p> |
| <ol> |
| <li>Listen for and enumerate camera devices.</li> |
| <li>Open device and connect listeners.</li> |
| <li>Configure outputs for target use case (such as still capture, recording, |
| etc.).</li> |
| <li>Create request(s) for target use case.</li> |
| <li>Capture/repeat requests and bursts.</li> |
| <li>Receive result metadata and image data.</li> |
| <li>When switching use cases, return to step 3.</li> |
| </ol> |
| <p><strong>HAL operation summary</strong></p> |
| <ul> |
| <li>Asynchronous requests for captures come from the framework.</li> |
| <li>HAL device must process requests in order. And for each request, produce |
| output result metadata, and one or more output image buffers.</li> |
| <li>First-in, first-out for requests and results, and for streams referenced by |
| subsequent requests. </li> |
| <li>Timestamps must be identical for all outputs from a given request, so that the |
| framework can match them together if needed. </li> |
| <li>All capture configuration and state (except for the 3A routines) is |
| encapsulated in the requests and results.</li> |
| </ul> |
| <img src="images/camera-hal-overview.png" alt="Camera HAL overview" id="figure3" /> |
| <p class="img-caption"> |
| <strong>Figure 3.</strong> Camera HAL overview |
| </p> |
| <h2 id="startup">Startup and expected operation sequence</h2> |
| <p>This section contains a detailed explanation of the steps expected when using |
| the camera API. Please see <a href="https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/camera3.h">platform/hardware/libhardware/include/hardware/camera3.h</a> for definitions of these structures and methods.</p> |
| <ol> |
| <li>Framework calls camera_module_t->common.open(), which returns a |
| hardware_device_t structure.</li> |
| <li>Framework inspects the hardware_device_t->version field, and instantiates the |
| appropriate handler for that version of the camera hardware device. In case |
| the version is CAMERA_DEVICE_API_VERSION_3_0, the device is cast to a |
| camera3_device_t.</li> |
| <li>Framework calls camera3_device_t->ops->initialize() with the framework |
| callback function pointers. This will only be called this one time after |
| open(), before any other functions in the ops structure are called.</li> |
| <li>The framework calls camera3_device_t->ops->configure_streams() with a list of |
| input/output streams to the HAL device.</li> |
| <li>The framework allocates gralloc buffers and calls |
| camera3_device_t->ops->register_stream_buffers() for at least one of the |
| output streams listed in configure_streams. The same stream is registered |
| only once.</li> |
| <li>The framework requests default settings for some number of use cases with |
| calls to camera3_device_t->ops->construct_default_request_settings(). This |
| may occur any time after step 3.</li> |
| <li>The framework constructs and sends the first capture request to the HAL with |
| settings based on one of the sets of default settings, and with at least one |
| output stream that has been registered earlier by the framework. This is sent |
| to the HAL with camera3_device_t->ops->process_capture_request(). The HAL |
| must block the return of this call until it is ready for the next request to |
| be sent.</li> |
| <li>The framework continues to submit requests, and possibly call |
| register_stream_buffers() for not-yet-registered streams, and call |
| construct_default_request_settings to get default settings buffers for other |
| use cases.</li> |
| <li>When the capture of a request begins (sensor starts exposing for the |
| capture), the HAL calls camera3_callback_ops_t->notify() with the SHUTTER |
| event, including the frame number and the timestamp for start of exposure. |
| This notify call must be made before the first call to |
| process_capture_result() for that frame number.</li> |
| <li>After some pipeline delay, the HAL begins to return completed captures to |
| the framework with camera3_callback_ops_t->process_capture_result(). These |
| are returned in the same order as the requests were submitted. Multiple |
| requests can be in flight at once, depending on the pipeline depth of the |
| camera HAL device.</li> |
| <li>After some time, the framework may stop submitting new requests, wait for |
| the existing captures to complete (all buffers filled, all results |
| returned), and then call configure_streams() again. This resets the camera |
| hardware and pipeline for a new set of input/output streams. Some streams |
| may be reused from the previous configuration; if these streams' buffers had |
| already been registered with the HAL, they will not be registered again. The |
| framework then continues from step 7, if at least one registered output |
| stream remains. (Otherwise, step 5 is required first.)</li> |
| <li>Alternatively, the framework may call camera3_device_t->common->close() to |
| end the camera session. This may be called at any time when no other calls |
| from the framework are active, although the call may block until all |
| in-flight captures have completed (all results returned, all buffers |
| filled). After the close call returns, no more calls to the |
| camera3_callback_ops_t functions are allowed from the HAL. Once the close() |
| call is underway, the framework may not call any other HAL device functions.</li> |
| <li>In case of an error or other asynchronous event, the HAL must call |
| camera3_callback_ops_t->notify() with the appropriate error/event message. |
| After returning from a fatal device-wide error notification, the HAL should |
| act as if close() had been called on it. However, the HAL must either cancel |
| or complete all outstanding captures before calling notify(), so that once |
| notify() is called with a fatal error, the framework will not receive |
| further callbacks from the device. Methods besides close() should return |
| -ENODEV or NULL after the notify() method returns from a fatal error |
| message.</li> |
| </ol> |
| <img src="images/camera-ops-flow.png" width="600" height="434" alt="Camera operations flow" id="figure4" /> |
| <p class="img-caption"> |
| <strong>Figure 4.</strong> Camera operational flow |
| </p> |
| <h2 id="ops-modes">Operational modes</h2> |
| <p>The camera 3 HAL device can implement one of two possible operational modes: |
| limited and full. Full support is expected from new higher-end devices. Limited |
| mode has hardware requirements roughly in line with those for a camera HAL |
| device v1 implementation, and is expected from older or inexpensive devices. |
| Full is a strict superset of limited, and they share the same essential |
| operational flow, as documented above.</p> |
| <p>The HAL must indicate its level of support with the |
| android.info.supportedHardwareLevel static metadata entry, with 0 indicating |
| limited mode, and 1 indicating full mode support.</p> |
| <p>Roughly speaking, limited-mode devices do not allow for application control of |
| capture settings (3A control only), high-rate capture of high-resolution images, |
| raw sensor readout, or support for YUV output streams above maximum recording |
| resolution (JPEG only for large images).<br/> |
| Here are the details of limited-mode behavior:</p> |
| <ul> |
| <li>Limited-mode devices do not need to implement accurate synchronization between |
| capture request settings and the actual image data captured. Instead, changes |
| to settings may take effect some time in the future, and possibly not for the |
| same output frame for each settings entry. Rapid changes in settings may |
| result in some settings never being used for a capture. However, captures that |
| include high-resolution output buffers ( > 1080p ) have to use the settings as |
| specified (but see below for processing rate).</li> |
| <li>Captures in limited mode that include high-resolution (> 1080p) output buffers |
| may block in process_capture_request() until all the output buffers have been |
| filled. A full-mode HAL device must process sequences of high-resolution |
| requests at the rate indicated in the static metadata for that pixel format. |
| The HAL must still call process_capture_result() to provide the output; the |
| framework must simply be prepared for process_capture_request() to block until |
| after process_capture_result() for that request completes for high-resolution |
| captures for limited-mode devices.</li> |
| <li>Limited-mode devices do not need to support most of the settings/result/static |
| info metadata. Only the following settings are expected to be consumed or |
| produced by a limited-mode HAL device: |
| <ul> |
| <li>android.control.aeAntibandingMode (controls)</li> |
| <li>android.control.aeExposureCompensation (controls)</li> |
| <li>android.control.aeLock (controls)</li> |
| <li>android.control.aeMode (controls)</li> |
| <li>[OFF means ON_FLASH_TORCH]</li> |
| <li>android.control.aeRegions (controls)</li> |
| <li>android.control.aeTargetFpsRange (controls)</li> |
| <li>android.control.afMode (controls)</li> |
| <li>[OFF means infinity focus]</li> |
| <li>android.control.afRegions (controls)</li> |
| <li>android.control.awbLock (controls)</li> |
| <li>android.control.awbMode (controls)</li> |
| <li>[OFF not supported]</li> |
| <li>android.control.awbRegions (controls)</li> |
| <li>android.control.captureIntent (controls)</li> |
| <li>android.control.effectMode (controls)</li> |
| <li>android.control.mode (controls)</li> |
| <li>[OFF not supported]</li> |
| <li>android.control.sceneMode (controls)</li> |
| <li>android.control.videoStabilizationMode (controls)</li> |
| <li>android.control.aeAvailableAntibandingModes (static)</li> |
| <li>android.control.aeAvailableModes (static)</li> |
| <li>android.control.aeAvailableTargetFpsRanges (static)</li> |
| <li>android.control.aeCompensationRange (static)</li> |
| <li>android.control.aeCompensationStep (static)</li> |
| <li>android.control.afAvailableModes (static)</li> |
| <li>android.control.availableEffects (static)</li> |
| <li>android.control.availableSceneModes (static)</li> |
| <li>android.control.availableVideoStabilizationModes (static)</li> |
| <li>android.control.awbAvailableModes (static)</li> |
| <li>android.control.maxRegions (static)</li> |
| <li>android.control.sceneModeOverrides (static)</li> |
| <li>android.control.aeRegions (dynamic)</li> |
| <li>android.control.aeState (dynamic)</li> |
| <li>android.control.afMode (dynamic)</li> |
| <li>android.control.afRegions (dynamic)</li> |
| <li>android.control.afState (dynamic)</li> |
| <li>android.control.awbMode (dynamic)</li> |
| <li>android.control.awbRegions (dynamic)</li> |
| <li>android.control.awbState (dynamic)</li> |
| <li>android.control.mode (dynamic)</li> |
| <li>android.flash.info.available (static)</li> |
| <li>android.info.supportedHardwareLevel (static)</li> |
| <li>android.jpeg.gpsCoordinates (controls)</li> |
| <li>android.jpeg.gpsProcessingMethod (controls)</li> |
| <li>android.jpeg.gpsTimestamp (controls)</li> |
| <li>android.jpeg.orientation (controls)</li> |
| <li>android.jpeg.quality (controls)</li> |
| <li>android.jpeg.thumbnailQuality (controls)</li> |
| <li>android.jpeg.thumbnailSize (controls)</li> |
| <li>android.jpeg.availableThumbnailSizes (static)</li> |
| <li>android.jpeg.maxSize (static)</li> |
| <li>android.jpeg.gpsCoordinates (dynamic)</li> |
| <li>android.jpeg.gpsProcessingMethod (dynamic)</li> |
| <li>android.jpeg.gpsTimestamp (dynamic)</li> |
| <li>android.jpeg.orientation (dynamic)</li> |
| <li>android.jpeg.quality (dynamic)</li> |
| <li>android.jpeg.size (dynamic)</li> |
| <li>android.jpeg.thumbnailQuality (dynamic)</li> |
| <li>android.jpeg.thumbnailSize (dynamic)</li> |
| <li>android.lens.info.minimumFocusDistance (static)</li> |
| <li>android.request.id (controls)</li> |
| <li>android.request.id (dynamic)</li> |
| <li>android.scaler.cropRegion (controls)</li> |
| <li>[ignores (x,y), assumes center-zoom]</li> |
| <li>android.scaler.availableFormats (static)</li> |
| <li>[RAW not supported]</li> |
| <li>android.scaler.availableJpegMinDurations (static)</li> |
| <li>android.scaler.availableJpegSizes (static)</li> |
| <li>android.scaler.availableMaxDigitalZoom (static)</li> |
| <li>android.scaler.availableProcessedMinDurations (static)</li> |
| <li>android.scaler.availableProcessedSizes (static)</li> |
| <li>[full resolution not supported]</li> |
| <li>android.scaler.maxDigitalZoom (static)</li> |
| <li>android.scaler.cropRegion (dynamic)</li> |
| <li>android.sensor.orientation (static)</li> |
| <li>android.sensor.timestamp (dynamic)</li> |
| <li>android.statistics.faceDetectMode (controls)</li> |
| <li>android.statistics.info.availableFaceDetectModes (static)</li> |
| <li>android.statistics.faceDetectMode (dynamic)</li> |
| <li>android.statistics.faceIds (dynamic)</li> |
| <li>android.statistics.faceLandmarks (dynamic)</li> |
| <li>android.statistics.faceRectangles (dynamic)</li> |
| <li>android.statistics.faceScores (dynamic)</li> |
| </ul> |
| </li> |
| </ul> |
| <h2 id="interaction">Interaction between the application capture request, 3A |
| control, and the processing pipeline</h2> |
| <p>Depending on the settings in the 3A control block, the camera pipeline ignores |
| some of the parameters in the application's capture request and uses the values |
| provided by the 3A control routines instead. For example, when auto-exposure is |
| active, the exposure time, frame duration, and sensitivity parameters of the |
| sensor are controlled by the platform 3A algorithm, and any app-specified values |
| are ignored. The values chosen for the frame by the 3A routines must be reported |
| in the output metadata. The following table describes the different modes of the |
| 3A control block and the properties that are controlled by these modes. See |
| the <a href="https://android.googlesource.com/platform/system/media/+/master/camera/docs/docs.html">platform/system/media/camera/docs/docs.html</a> file for definitions of these properties.</p> |
| <table> |
| <tr> |
| <th>Parameter</th> |
| <th>State</th> |
| <th>Properties controlled</th> |
| </tr> |
| <tr> |
| <td>android.control.aeMode</td> |
| <td>OFF</td> |
| <td>None</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>ON</td> |
| <td>android.sensor.exposureTime |
| android.sensor.frameDuration |
| android.sensor.sensitivity |
| android.lens.aperture (if supported) |
| android.lens.filterDensity (if supported)</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>ON_AUTO_FLASH</td> |
| <td>Everything is ON, plus android.flash.firingPower, android.flash.firingTime, and android.flash.mode</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>ON_ALWAYS_FLASH</td> |
| <td>Same as ON_AUTO_FLASH</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>ON_AUTO_FLASH_RED_EYE</td> |
| <td>Same as ON_AUTO_FLASH</td> |
| </tr> |
| <tr> |
| <td>android.control.awbMode</td> |
| <td>OFF</td> |
| <td>None</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>WHITE_BALANCE_*</td> |
| <td>android.colorCorrection.transform. Platform-specific adjustments if android.colorCorrection.mode is FAST or HIGH_QUALITY.</td> |
| </tr> |
| <tr> |
| <td>android.control.afMode</td> |
| <td>OFF</td> |
| <td>None</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>FOCUS_MODE_*</td> |
| <td>android.lens.focusDistance</td> |
| </tr> |
| <tr> |
| <td>android.control.videoStabilization</td> |
| <td>OFF</td> |
| <td>None</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>ON</td> |
| <td>Can adjust android.scaler.cropRegion to implement video stabilization</td> |
| </tr> |
| <tr> |
| <td>android.control.mode</td> |
| <td>OFF</td> |
| <td>AE, AWB, and AF are disabled</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>AUTO</td> |
| <td>Individual AE, AWB, and AF settings are used</td> |
| </tr> |
| <tr> |
| <td></td> |
| <td>SCENE_MODE_*</td> |
| <td>Can override all parameters listed above. Individual 3A controls are disabled.</td> |
| </tr> |
| </table> |
| <p>The controls exposed for the 3A algorithm mostly map 1:1 to the old API's |
| parameters (such as exposure compensation, scene mode, or white balance mode).<br/> |
| The controls in the Image Processing block in Figure 2</a> all |
| operate on a similar principle, and generally each block has three modes:</p> |
| <ul> |
| <li>OFF: This processing block is disabled. The demosaic, color correction, and |
| tone curve adjustment blocks cannot be disabled.</li> |
| <li>FAST: In this mode, the processing block may not slow down the output frame |
| rate compared to OFF mode, but should otherwise produce the best-quality |
| output it can given that restriction. Typically, this would be used for |
| preview or video recording modes, or burst capture for still images. On some |
| devices, this may be equivalent to OFF mode (no processing can be done without |
| slowing down the frame rate), and on some devices, this may be equivalent to |
| HIGH_QUALITY mode (best quality still does not slow down frame rate).</li> |
| <li>HIGHQUALITY: In this mode, the processing block should produce the best |
| quality result possible, slowing down the output frame rate as needed. |
| Typically, this would be used for high-quality still capture. Some blocks |
| include a manual control which can be optionally selected instead of FAST or |
| HIGHQUALITY. For example, the color correction block supports a color |
| transform matrix, while the tone curve adjustment supports an arbitrary global |
| tone mapping curve.</li> |
| </ul> |
| <p>The maximum frame rate that can be supported by a camera subsystem is a function |
| of many factors:</p> |
| <ul> |
| <li>Requested resolutions of output image streams</li> |
| <li>Availability of binning / skipping modes on the imager</li> |
| <li>The bandwidth of the imager interface</li> |
| <li>The bandwidth of the various ISP processing blocks</li> |
| </ul> |
| <p>Since these factors can vary greatly between different ISPs and sensors, the |
| camera HAL interface tries to abstract the bandwidth restrictions into as simple |
| model as possible. The model presented has the following characteristics:</p> |
| <ul> |
| <li>The image sensor is always configured to output the smallest resolution |
| possible given the application's requested output stream sizes. The smallest |
| resolution is defined as being at least as large as the largest requested |
| output stream size.</li> |
| <li>Since any request may use any or all the currently configured output streams, |
| the sensor and ISP must be configured to support scaling a single capture to |
| all the streams at the same time. </li> |
| <li>JPEG streams act like processed YUV streams for requests for which they are |
| not included; in requests in which they are directly referenced, they act as |
| JPEG streams.</li> |
| <li>The JPEG processor can run concurrently to the rest of the camera pipeline but |
| cannot process more than one capture at a time.</li> |
| </ul> |
