page.title=Camera HAL Testing Checklist
@jd:body

<!--
    Copyright 2016 The Android Open Source Project

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-->
<div id="qv-wrapper">
  <div id="qv">
    <h2>In this document</h2>
    <ol id="auto-toc">
    </ol>
  </div>
</div>

<p>This document lists all tests available for evaluating the Android camera
hardware abstraction layer (HAL). It is intended for OEMs and application
processor (AP) vendors so they may ensure proper implementation of the camera
HAL with minimum defects. Although this is a voluntary addition to the Android
Compatibility Test Suite (CTS), it greatly increases camera test coverage and
will certainly identify potential bugs.</p>

<p>By passing these tests, original equipment manufacturers (OEM) validate whether
they have properly integrated the latest Android camera hardware abstraction
layer (HAL) 3.2 interfaces. When conforming with all items in the checklist, a
device implementation may be considered <em>full</em> with respect to the new Android
Camera HAL interfaces. This will in turn enable a device to properly support
the new <code>android.hardware.camera2</code> package that camera apps build upon.</p>

<h2 id=camera_hal_3_2_specification>[  ] 1. Camera HAL 3.2 specification</h2>

<p>The Android Camera HAL 3.2 specification is the authoritative source of
information on what devices must satisfy; the document here provides a summary
of all tests that can be used as a checklist. Camera HAL implementers (e.g. AP
vendors) should go through the HAL 3.2 specification line-by-line and ensure
their devices conform to it.</p>

<p>The current HAL 3.2 specification is defined in these files within the L
generic Android Platform Development Kit (PDK):</p>

<ul>
  <li><em>Camera HAL 3.x interface and spec</em>: <code><a
href="https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/camera3.h">hardware/libhardware/include/hardware/camera3.h</a></code>,
<code><a
href="https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/camera_common.h">hardware/libhardware/include/hardware/camera_common.h</a></code>
  <li><em>Camera HAL 3.x metadata spec</em>: <code><a
href="https://android.googlesource.com/platform/system/media/+/master/camera/docs/docs.html">system/media/camera/docs/docs.html</a></code>
  <li><em>HAL pixel format interface and spec</em>: <code><a
href="https://android.googlesource.com/platform/system/core/+/master/include/system/graphics.h">system/core/include/system/graphics.h</a></code>
</ul>

<h2 id=camera_test_types>[  ] 2. Camera test types</h2>

<p>Here are the primary types of tests available for the latest Android camera
along with references to associated instructions below:</p>

<ul>
  <li><em><a href="#native_tests">Native</a>:</em> Tests that directly test the camera HAL interface
  <li><em><a href="#cts_tests">Compatibility Test Suite (CTS)</a></em>: Standard, automated Android
tests to ensure device compatibility - see the <a
href="{@docRoot}compatibility/cts/index.html">CTS introduction</a> and the <a
href="{@docRoot}devices/tech/test_infra/tradefed/index.html">Trade Federation
Overview</a>
  <li><em><a href="#its_tests">Image Test Suite (ITS)</a>:</em> Manually run tests to ensure image
correctness - see the top-level and
test-specific <code>README</code> files and tutorial.py for setup instructions
  <li><em><a href="#manual_tests_with_aosp_camera_app">Manual tests with the
Android Open Source Project (AOSP) Camera App</a>:</em> User-like testing of
common camera functions
  <li><em><a href="#manual_testingcam_tests">Manual TestingCam tests</a>:</em>
Run from the source in <code>pdk/apps/TestingCamera/</code>
  <li><em><a href="#manual_testingcam2_tests">Manual TestingCam2.1
tests</a>:</em> Run from the source in <code>pdk/apps/TestingCamera2/</code>
</ul>

<p>All of these test types are described in detail below. These tests are
presented in the chronological order in which OEMs are expected to execute
them.</p>

<p>For instance, if a device fails the native tests, it will assuredly fail the
subsequent Compatibility Test Suite (CTS) tests. And if a device fails CTS,
there is little use in proceeding to the Image Test Suite (ITS). We recommend
addressing failures in each test type before proceeding to the next set of
tests.</p>

<h2 id=native_tests>[  ] 3. Native tests</h2>

<p>These tests directly test the camera HAL interface.</p>

<p>The starting path for Camera native tests is:
<code>platform/hardware/libhardware</code></p>

<p>To set up these tests:</p>

<pre>
$ cd hardware/libhardware/tests/camera*/
$ mm
$ adb remount; adb sync
</pre>

<h3 id=hal_3_x_tests>[  ] 3.1. HAL 3.x tests</h3>

<p>Find these camera tests under:
<code>hardware/libhardware/tests/camera3/*</code></p>

<p>To run all tests:</p>

<pre>
$ adb shell /data/nativetest/camera3_test/camera3_test
</pre>

<p>You receive an <strong>OK</strong> for each passed test. Errors are logged
at the end of output along with a summary of tests passed.</p>

<h3 id=hal_2_3_tests>[  ] 3.2. HAL 2/3 tests</h3>

<p>Find these camera tests under:
<code>hardware/libhardware/tests/camera2/*</code></p>

<p>To run all tests:</p>

<pre>
$ adb shell /data/nativetest/camera3_test/camera3_test
</pre>

<p>To run a single test, pass the <code>--gtest_filter</code> argument and the
test name, like so:</p>

<pre>
$ adb shell /data/nativetest/camera2_test/camera2_test \
--gtest_filter=Camera2Test.OpenClose
</pre>

<p>To run a subset of tests, use a wildcard with the
<code>--gtest_filter</code> argument, like so:</p>

<pre>
$ adb shell /data/nativetest/camera2_test/camera2_test \
--gtest_filter=Camera2Test.*
</pre>

<h3 id=3_tests_that_interact_with_the_camera_service>[  ] 3.3. Tests that
interact with the camera service</h3>

<p>Find these camera tests under: <code>frameworks/av/camera/tests/*</code></p>

<h3 id=camera_metadata_tests>[  ] 3.4. Camera metadata tests</h3>

<p>Find these camera tests under: <code>system/media/camera/tests/*</code></p>

<h2 id=cts_tests>[  ] 4. Compatibility Test Suite (CTS) tests</h2>

<p>Camera Android Compatibility Test Suite (CTS) tests focus upon device
compatibility. They do not require a specific test environment (the field of
view or FOV CTS Verifier test being the lone exception).</p>

<p>The starting path for Camera CTS tests is: <code>platform/cts</code></p>

<p>See the <a href="{@docRoot}compatibility/cts/index.html">CTS
introduction</a> and its subpages for general instructions on running CTS.</p>

<h3 id=cts_tests_for_the_android_hardware_camera_api>[  ] 4.1. CTS tests for
the <code>android.hardware.Camera</code> API</h3>

<p>Find these camera tests under <code>cts/tests/tests/</code>:</p>

<ul>
  <li><code>hardware/src/android/hardware/cts/CameraTest.java</code>
  <li><code>hardware/src/android/hardware/cts/CameraGLTest.java</code>
  <li><code>hardware/src/android/hardware/cts/Camera_SizeTest.java</code>
  <li><code>permission/src/android/permission/cts/CameraPermissionTest.java</code>
</ul>

<h3 id=cts_tests_for_the_android_hardware_camera2_api>[  ] 4.2. CTS tests for
the <code>android.hardware.camera2</code> API</h3>

<p>Find these camera tests under <code>cts/tests/tests/</code>:</p>

<ul>
  <li><code>hardware/src/android/hardware/camera2/cts/*</code>
  <li><code>permission/src/android/permission/cts/Camera2PermissionTest.java</code>
</ul>

<h3 id=cts_verifier_camera_tests>[  ] 4.3. CTS Verifier camera tests</h3>

<p>Find these camera tests under:
<code>cts/apps/CtsVerifier/src/com/android/cts/verifier/camera/*</code></p>

<h2 id=its_tests>[  ] 5. Image Test Suite (ITS) tests</h2>

<p>The CameraITS tests focus upon image correctness. These Python scripts are
manually run on a workstation with the Android device connected over USB. The
workstation can run any operating system as long as it has the requisite Python
2.7 environment.</p>

<p class="note">Since ITS is a CTS Verifier subtest, start CTS Verifier and the
ITS subtest before running the python scripts so they have processes with which
to communicate.</p>

<p>The CameraITS infrastructure and tests are located under: <code>cts/apps/CameraITS</code></p>

<p>See the latest <code>README</code> file in this top-level folder for
instructions on how to set up and run the tests. For setup: <code>make
cts</code></p>

<pre>
$ extract root/out/host/linux-x86/cts-verfier/android-cts-verifier.zip
$ cd android-cts-verifier
$ adb install -r CtsVerifier.apk
$ cd CameraITS
</pre>

<p>See <code>tutorial.py</code> in the <code>tests</code> subdirectory for a
walkthrough of the script's use.  Each test resides in a corresponding
<code>tests/scene<#></code> subdirectory. See the <code>README</code> file in
each subdirectory for specific test instructions.</p>

<p>You will need a simple physical environment with a specific, reusable target
such as a white wall, grey card, and desk lamp. The Android device is mounted
on a tripod and its camera functions are exercised by the scripts. Most tests
are pass or fail but some offer metrics, as well.</p>

<p>These tests are works-in-progress and are not yet comprehensive enough for
full automated pass/fail validation of the camera HAL. However, these scripts
do test scenarios that are not tested in CTS and are an important component of
the overall HAL 3.2 test plan.</p>

<h3 id=its_tests_on_scene_0_plain>[  ] 5.1. ITS tests on scene 0 (plain)</h3>

<p>This test requires no specific setup. Pass all of the tests in the
<code>tests/scene0</code> folder, for all cameras (back + front + any
others).</p>

<h3 id=its_tests_on_scene_1_grey_card>[  ] 5.2. ITS tests on scene 1 (grey card)</h3>

<p>Pass all of the tests in the <code>tests/scene1</code> folder, for all
cameras (back + front + any others). The <code>tests/scene1/README</code> file
describes the scene setup.</p>

<h3 id=its_tests_on_scene_2_camera_lab>[  ] 5.3. ITS tests on scene 2 (camera lab)</h3>

<p>Pass all of the tests in the <code>tests/scene2</code> folder, for all
cameras (back + front + any others). The <code>tests/scene2/README</code> file
describes the scene setup.</p>

<h2 id=manual_tests_with_aosp_camera_app>[  ] 6. Manual tests with the AOSP App</h2>

<h3 id=camera_mode_aosp_camera_app>[  ] 6.1. Camera mode</h3>

<p>For all cameras on the device (front, back, and any others), verify:</p>

<ol>
  <li>Images can be captured and reviewed on the device, and the images look good
with no obvious problems.
  <li>Tap-to-focus, continuous autofocus, macro focus, infinity focus, AWB, and AEC
are all reliable.
  <li>Tap-to-focus, continuous autofocus, AWB, and AEC are reliable when using
digital zoom (during capture).
  <li>Flash settings (on/off/auto) are reliable and synchronize well with the 3As.
</ol>

<h3 id=video_mode_aosp_camera_app>[  ] 6.2. Video mode</h3>

<p>For all cameras on the device (front, back, and any others), verify:</p>

<ol>
  <li>Videos can be captured and reviewed on the device, and the videos look good
with no obvious problems.
  <li>Capturing a snapshot while in the middle of recording a video works.
  <li>Tap-to-focus, continuous autofocus, macro focus, infinity focus, AWB, and AEC
are all reliable.
  <li>Tap-to-focus, continuous autofocus, AWB, and AEC are reliable when using
digital zoom (during capture).
  <li>Torch settings (on/off) are reliable and synchronize well with the 3As.
</ol>

<h3 id=camera_settings_resolution>[  ] 6.3. Camera settings: resolution</h3>

<p>For all cameras on the device (front, back, and any others), and for all
resolutions available in the menu, verify that correct resolution settings are
returned and applied for:</p>

<ul>
  <li>Camera mode
  <li>Video mode
  <li>LensBlur
  <li>PhotoSphere
  <li>Panorama
</ul>

<h3 id=camera_settings_exposure_compensation>[  ] 6.4. Camera settings:
exposure compensation</h3>

<p>Verify that exposure compensation is applied (at +2 and -2).</p>

<h3 id=photosphere>[  ] 6.5. PhotoSphere</h3>

<p>Capture full 360-degree PhotoSphere images shot with each of the front and
rear cameras. Verify all of the individual frames are focused at infinity and the
exposure and white balance match between shots.</p>

<h3 id=panorama>[  ] 6.6. Panorama</h3>

<p>Capture vertical, horizontal, and wide-angle panoramas (with both front and
rear cameras), and verify all of the individual frames are focused at infinity
and the exposure and white balance matches between shots.</p>

<h3 id=lensblur>[  ] 6.7. LensBlur</h3>

<p>Capture a LensBlur image with both front and rear cameras, and verify
refocusing to different depths (while reviewing the captured shots) works.</p>

<p>Also verify tap-to-focus, continuous autofocus, AWB, and AEC are reliable in
this mode.</p>

<h2 id=media_framework_tests>[  ] 7. Media Framework tests</h2>

<p>Pass all of the camera-related media tests in MediaFrameworkTest. Please note,
these tests require the mediaframeworktest.apk be installed on the Android
device. You will need to <code>make mediaframeworktest</code> and then use adb
to install the resulting .apk. Example commands are included below.</p>

<p>The starting path for Camera-related media framework tests is:
<code>platform/frameworks/base</code></p>

<p>Find the source code for the tests here:
<code>frameworks/base/media/tests/MediaFrameworkTest</code></p>

<p>To set up these tests:</p>

<pre>
$ make mediaframeworktest
$ adb install out/target/product/<em>&lt;name></em>/data/app/mediaframeworktest.apk
</pre>

<p>Where the <em><code><name></code></em> variable represents the directory
containing the vendor's product.</p>

<p>Find all of the tests in the following directory or its subdirectories:</p>

<pre>
frameworks/base/media/tests/MediaFrameworkTest/src/com/android/mediaframeworktest
</pre>

<p>Each subdirectory represents a class of tests:</p>

<ul>
  <li><code>functional/</code>
  <li><code>integration/</code>
  <li><code>performance/</code>
  <li><code>power/</code>
  <li><code>stress/</code>
  <li><code>unit/</code>
</ul>

<h3 id=running_media_framework_tests>[  ] 7.1. Running Media Framework tests</h3>

<p>To see all of the available tests::</p>

<pre>
$ adb shell pm list instrumentation
</pre>

<p>This will yield results resembling:</p>

<pre>
instrumentation:com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
(target=com.android.mediaframeworktest)
instrumentation:com.android.mediaframeworktest/.MediaRecorderStressTestRunner
(target=com.android.mediaframeworktest)
instrumentation:com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner
(target=com.android.mediaframeworktest)
instrumentation:com.android.mediaframeworktest/.MediaFrameworkPowerTestRunner
(target=com.android.mediaframeworktest)
</pre>

<p>Identify and extract the component (between <code>instrumentation:</code>
and <code>(target=com.android.mediaframeworktest) </code>from each test line.
The component is composed of the target package name
(<code>com.android.mediaframeworktest</code>) and the test runner name
(<code>MediaFramework<type>TestRunner</code>).</p>

<p>For instance:</p>

<pre>
com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
com.android.mediaframeworktest/.MediaRecorderStressTestRunner
com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner
com.android.mediaframeworktest/.MediaFrameworkPowerTestRunner
</pre>

<p>You may then pass each component to <code>adb shell am instrument</code> like so:</p>

<pre>
$ adb shell am instrument -w &lt;component.name&gt;
</pre>

<p>Where the &lt;component.name&gt; equals the extracted value above. For example:</p>

<pre>
$ adb shell am instrument -w \
com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
</pre>

<p>Please note, while the class path is the Java package + class name, the
instrumentation package isn't necessarily the same as the Java package. Make
sure you use the AndroidManifest.xml package when concatenating the component
name, not the Java package in which the test runner class resides.</p>

<p>To run a single class of tests, pass the -e class <test-class> argument, like
so:</p>

<pre>
$ adb shell am instrument -e class \
com.android.mediaframeworktest.integration.CameraBinderTest -w \
com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
</pre>

<p>To run only a single method in a test class, append a pound (#) sign and the
method name (in this case, <code>testConnectPro</code>) to the class name, like so:</p>

<pre>
$ adb shell am instrument -e class \
'com.android.mediaframeworktest.integration.CameraBinderTest#testConnectPro' -w
\ com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
</pre>

<h3 id=media_settings_functional_tests>[  ] 7.2. Media settings functional tests</h3>

<p>Here is an example run of a functional test. This test verifies the basic
functionality of different combinations of camera settings. (ie, Flash,
exposure, WB, scene, picture size and geoTag)</p>

<p>Run the test command:</p>
<pre>
$ adb shell am instrument -w -r  -e delay_msec 15 -e log true -e class \
com.android.mediaframeworktest.functional.camera.CameraPairwiseTest \
com.android.mediaframeworktest/com.android.mediaframeworktest.CameraStressTestRunner
</pre>

<h3 id=media_integration_tests>[  ] 7.3. Media integration tests</h3>

<p>Here is an example run of an integration test, in this case
mediaframeworktest/integration/CameraBinderTest.java and
mediaframeworktest/CameraStressTestRunner.java:</p>

<pre>
$ adb  shell am instrument -e class \ 'com.android.mediaframeworktest.<strong>integration</strong>.<strong>CameraBinderTest'</strong> -w \ 'com.android.mediaframeworktest/.<strong>CameraStressTestRunner'</strong>
</pre>

<p>If successful, this results in output resembling:</p>

<pre>
-----

com.android.mediaframeworktest.integration.CameraBinderTest:...........
Test results for CameraStressTestRunner=...........
Time: 3.328

OK (11 tests)

-----
</pre>

<h3 id=media_performance_tests>[  ] 7.4. Media performance tests</h3>

<p>This preview memory test will open and release the camera preview for 200
times. In each 20 iterations, the snapshot of ps mediaserver will be recorded
and it will compare the memory usage different after 200 iterations. Test will
fail If the difference is greater than 150kM.</p>

<p>Run the test command:</p>
<pre>
$ adb shell am instrument -w -r  -e class \
com.android.mediaframeworktest.performance.MediaPlayerPerformance#testCameraPreviewMemoryUsage
\ com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner
</pre>

<p>More detailed output can be found in:
<code>/sdcard/mediaMemOutput.txt</code></p>

<h3 id=media_unit_tests>[  ] 7.5. Media unit tests</h3>

<p>The commands to run unit tests are all similar. For example, for
CameraMetadataTest.java, the command would be:</p>

<pre>
$ adb  shell am instrument -e class \
'com.android.mediaframeworktest.unit.CameraMetadataTest' -w \
'com.android.mediaframeworktest/.CameraStressTestRunner'
</pre>

<h3 id=media_stress_tests>[  ] 7.6. Media stress tests</h3>

<p>This test is to stress out the camera image capture and video recording.</p>

<p>Run the test command:</p>

<pre>
$ adb shell am instrument -w
com.google.android.camera.tests/com.android.camera.stress.CameraStressTestRunner
</pre>

<p>All tests should pass.</p>

<h2 id=manual_testingcam_tests>[  ] 8. Manual TestingCam tests</h2>

<p>The TestingCam app should be run manually with the following checks performed.
The source for TestingCam is here: <code>pdk/apps/TestingCamera/</code></p>

<h3 id=infinity_focus_with_camera_tilt>[  ] 8.1. Infinity focus with camera tilt</h3>

<p>Start TestingCam, turn on preview, and ensure that autofocus mode is set to
infinity. Using the <strong>Take picture</strong> button, capture shots of
distant subjects (at least 10m away) with the camera pointed horizontally,
upwards (close to vertical), and downwards (close to vertical); an example of
the upwards shot could be high leaves/branches of a tree from beneath and an
example of the downwards shot could be the street as seen from the roof of a
building. In all cases, the distant subject should be sharp and in focus. Save
and view the shots in the gallery view so that you can zoom in and inspect the
sharpness more easily.</p>

<p>Note that for a camera with a VCM actuator to pass this test, it will require
either a closed-loop AF control system, or it will need some sort of SW
correction based on using accelerometer data to determine camera orientation.
Reliable factory calibration of the lens infinity position will also be needed.</p>

<h2 id=manual_testingcam2_tests>[  ] 9. Manual TestingCam2 tests</h2>

<p>The TestingCam2 app should be run manually, with the following checks
performed. The source for TestingCam2 is here: <code>pdk/apps/TestingCamera2/</code></p>

<h3 id=9_1_jpeg_capture>[  ] 9.1. JPEG capture</h3>

<p>Start TestingCam2, and press the <strong>JPEG</strong> button. The image
that appears to the right of the viewfinder image should appear the same as the
viewfinder, including having the same orientation.</p>