en/compatibility/vts/hal-testability.html - platform/docs/source.android.com - Git at Google

 <html devsite>
 <head>
   <title>HAL Testability Check</title>
   <meta name="project_path" value="/_project.yaml">
   <meta name="book_path" value="/_book.yaml">
 </head>
 <body>
   {% include "_versions.html" %}
   <!--
       Copyright 2018 The Android Open Source Project

       Licensed under the Apache License, Version 2.0 (the "License");
       you may not use this file except in compliance with the License.
       You may obtain a copy of the License at

           http://www.apache.org/licenses/LICENSE-2.0

       Unless required by applicable law or agreed to in writing, software
       distributed under the License is distributed on an "AS IS" BASIS,
       WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
       See the License for the specific language governing permissions and
       limitations under the License.
   -->

 <p>
   The Android {{ androidPVersionNumber }} Vendor Test Suite (VTS) supports a
   runtime method for using the device configuration to identify which VTS tests
   should be skipped for that device target.
 </p>

 <h3 id="vts-test-flexibility">VTS test flexibility</h3>

 <p>
   As of Android 8.0, VTS tests are required for all devices launched with
   Android 8.0 and higher. However, not all VTS tests apply to all device
   targets. For example:
 </p>

 <ul>
   <li>If a specific device does not support a testing HAL (e.g. IR), VTS does
   not need to run tests for that HAL test against that device target.</li>
   <li>If several devices share the same SoC and vendor image but have
   different hardware functionalities, VTS must determine whether a test
   should be run or be skipped for a specific device target.</li>
 </ul>

 <h3 id="vts-test-types">VTS test types</h3>

 <p>
   VTS includes the following test types:
 </p>

 <ul>
   <li><strong>Compliance</strong> tests ensure compatibility between framework
     and vendor partitions. These tests are required to be run (and pass) on
   devices launching with Android 8.0 or higher.</li>
   <li><strong>Non-compliance</strong> tests help vendors to improve product
   quality (performance/fuzzing etc.). These tests are optional for vendors.</li>
 </ul>

 <p>
   Whether a test is a compliance test or not depends on which plan it belongs
   to. Tests that run with
   <a href="https://android.googlesource.com/platform/test/vts/+/master/tools/vts-tradefed/res/config/vts.xml" class="external">
   VTS plan</a> are considered compliance tests.
 </p>

 <h2 id="determine-supported-hals">Determining supported HALs</h2>

 <p>
   VTS can use the following files to determine if the device target supports a
   specific HAL:
 </p>

 <ul>
   <li><code>/system/compatibility_matrix.xml</code>. Claims the HAL instances
   required by the framework. Example:
 <pre class="prettyprint">
 &lt;hal format="hidl" optional="true"&gt;
     &lt;name&gt;android.hardware.vibrator&lt;/name&gt;
     &lt;version&gt;1.0-1&lt;/version&gt;
     &lt;interface&gt;
        &lt;name&gt;IVibrator&lt;/name&gt;
        &lt;instance&gt;default&lt;/instance&gt;
     &lt;/interface&gt;
 &lt;/hal&gt;
 </pre>
   <ul>
     <li>The <code>optional</code> attribute indicates if the HAL is strictly
     required by the framework.</li>
     <li>The file may contain multiple entries for the same HAL (with same name)
     but with different version and interfaces.</li>
     <li>The file may contain multiple <code>version</code> configurations for
     the same entry, indicating the framework can work with different versions.
     </li>
     <li><code>version1.0-1</code> means the framework can work with the lowest
     version 1.0, and does not require a version higher than 1.1.</li>
   </ul>
   </li>
   <li>Device <code>manifest.xml</code>. Claims the HAL instances provided by the
   vendor. Example:
 <pre class="prettyprint">
 &lt;hal format="hidl"&gt;
     &lt;name&gt;android.hardware.vibrator&lt;/name&gt;
     &lt;transport&gt;hwbinder&lt;/transport&gt;
     &lt;version&gt;1.2&lt;/version&gt;
     &lt;interface&gt;
         &lt;name&gt;IVibrator&lt;/name&gt;
        &lt;instance&gt;default&lt;/instance&gt;
     &lt;/interface&gt;
 &lt;/hal&gt;
 </pre>
   <ul>
     <li>The file may contain multiple entries for the same HAL (with same name)
      but with different version and interfaces.</li>
     <li>If the file contains only a single <code>version</code> configuration
     for an entry, <code>version1.2</code> means the vendor supports all versions
     from 1.0~1.2.</li>
   </ul>
   </li>
   <li><strong>lshal</strong>. A tool on device that shows runtime info about
   the HAL services registered with the <code>hwservicemanager</code>. Example:
 <pre class="prettyprint">
 android.hardware.vibrator@1.0::IVibrator/default
 </pre>
   <br><code>lshal</code> also shows all the HALs that with passthrough
   implementations (i.e having the corresponding <code>-impl.so</code> file on
   the device). Example:
 <pre class="prettyprint">
 android.hardware.nfc@1.0::I*/* (/vendor/lib/hw/)
 android.hardware.nfc@1.0::I*/* (/vendor/lib64/hw/)
 </pre>
   </li>
   </ul>

 <h2 id="compliance-tests">Compliance tests</h2>

 <p>
   For compliance tests, VTS relies on the vendor manifest to determine (and
   test) all HAL instances provided by the device. Decision flow:
 </p>

 <p>
   <img src="images/testability-check-compliance.png"
   alt="Testability check for compliance"
   title="Testability check for compliance">
 </p>
 <figcaption>
   <strong>Figure 1.</strong> Testability check for VTS compliance tests
 </figcaption>

 <h2 id="non-compliance-tests">Non-compliance tests</h2>

 <p>
   For non-compliance tests, VTS relies on the vendor manifest and
   <code>lshal</code> outputs to determine (and test) the experimental HALs not
   claimed in the <code>manifest.xml</code> file. Decision flow:
 </p>

 <p>
   <img src="images/testability-check-non-compliance.png"
   alt="Testability check for non-compliance"
   title="Testability check for non-compliance">
 <figcaption>
   <strong>Figure 2.</strong> Testability check for VTS non-compliance
   tests
 </figcaption>

 <h2 id="locating-the-vendor-manifest">Locating the vendor manifest</h2>

 <p>
   VTS checks for the vendor <code>manifest.xml</code> file in the following
   places in the following order:
 </p>

 <ol>
   <li><code>/vendor/etc/vintf/manifest.xml</code> + ODM manifest (If same HAL
   is defined in both places, ODM manifest overrides the one in
   <code>/vendor/etc/vintf/manifest.xml</code>)</li>
   <li><code>/vendor/etc/vintf/manifest.xml</code></li>
   <li>ODM <code>manifest.xml</code> file, loaded from the following files in
   the following order:
   <ol>
   <li><code>/odm/etc/vintf/manifest_$(ro.boot.product.hardware.sku).xml</code>
   </li>
   <li><code>/odm/etc/vintf/manifest.xml</code></li>
   <li><code>/odm/etc/manifest_$(ro.boot.product.hardware.sku).xml</code></li>
   <li><code>/odm/etc/manifest.xml</code></li>
   <li><code>/vendor/manifest.xml</code></li>
   </ol>
   </li>
 </ol>

 <h2 id="vts-testability-checker">VTS testability checker</h2>

 <p>
   The
   <code><a href="https://android.googlesource.com/platform/test/vts/+/master/utils/native/testability_checker/?q=vts_testability&amp;g=0" class="external">
   vts_testibility_checker</a></code> is a binary packaged with VTS and used by
   VTS test framework at runtime to determine whether a given HAL test is
   testable or not. It is based on
   <code><a href="https://android.googlesource.com/platform/system/libvintf/+/master" class="external">libvintf</a></code>
   to load and parse the vendor manifest file and implements the decision flow
   described in the previous section.
 </p>

 <p>
   To use <code>vts_testability_check</code>:
 </p>

 <ul>
   <li>For a compliance test:
 <pre class="prettyprint">
 vts_testability_check -c -b &lt;bitness&gt;  &lt;hal@version&gt;
 </pre>
   </li>
   <li>For a non-compliance test:
 <pre class="prettyprint">
 vts_testability_check -b &lt;bitness&gt;  &lt;hal@version&gt;
 </pre>
   </li>
 </ul>

 <p>
   The output of <code>vts_testability_check</code> uses the following json
   format:
 </p>

 <pre class="prettyprint">
 {testable: &lt;True/False&gt; Instances: &lt;list of instance names of HAL service&gt;}
 </pre>

 <h2 id="determining-accessed-hals">Determining accessed HALs</h2>

 <p>
   To determine which HALs are accessed by VTS tests, ensure that each HAL test
   uses the
   <code><a href="https://android.googlesource.com/platform/test/vts/+/master/runners/target/vts_hal_hidl_target/VtsHalHidlTargetTestEnvBase.h" class="external">VtsHalHidlTargetTestEnvBase</a></code>
   template to register the HAL(s) accessed in the test. The VTS testing
   framework can then extract the registered HALs when pre-processing the test.
 </p>

 <p>
   For compliance tests, you can also check
   <code>/system/etc/vintf/manifest.xml</code>. If a HAL is defined here, VTS
   should test it. (For the HAL services provided by the system (e.g.
   <code>graphics.composer/vr</code>), the HALs are declared in
   <code>/system/manifest.xml</code>.)
 </p>

 </body>
 </html>
	<html devsite>
	<head>
	<title>HAL Testability Check</title>
	<meta name="project_path" value="/_project.yaml">
	<meta name="book_path" value="/_book.yaml">
	</head>
	<body>
	{% include "_versions.html" %}
	<!--
	Copyright 2018 The Android Open Source Project

	Licensed under the Apache License, Version 2.0 (the "License");
	you may not use this file except in compliance with the License.
	You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing, software
	distributed under the License is distributed on an "AS IS" BASIS,
	WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	See the License for the specific language governing permissions and
	limitations under the License.
	-->

	<p>
	The Android {{ androidPVersionNumber }} Vendor Test Suite (VTS) supports a
	runtime method for using the device configuration to identify which VTS tests
	should be skipped for that device target.
	</p>

	<h3 id="vts-test-flexibility">VTS test flexibility</h3>

	<p>
	As of Android 8.0, VTS tests are required for all devices launched with
	Android 8.0 and higher. However, not all VTS tests apply to all device
	targets. For example:
	</p>

	<ul>
	<li>If a specific device does not support a testing HAL (e.g. IR), VTS does
	not need to run tests for that HAL test against that device target.</li>
	<li>If several devices share the same SoC and vendor image but have
	different hardware functionalities, VTS must determine whether a test
	should be run or be skipped for a specific device target.</li>
	</ul>

	<h3 id="vts-test-types">VTS test types</h3>

	<p>
	VTS includes the following test types:
	</p>

	<ul>
	<li><strong>Compliance</strong> tests ensure compatibility between framework
	and vendor partitions. These tests are required to be run (and pass) on
	devices launching with Android 8.0 or higher.</li>
	<li><strong>Non-compliance</strong> tests help vendors to improve product
	quality (performance/fuzzing etc.). These tests are optional for vendors.</li>
	</ul>

	<p>
	Whether a test is a compliance test or not depends on which plan it belongs
	to. Tests that run with
	<a href="https://android.googlesource.com/platform/test/vts/+/master/tools/vts-tradefed/res/config/vts.xml" class="external">
	VTS plan</a> are considered compliance tests.
	</p>

	<h2 id="determine-supported-hals">Determining supported HALs</h2>

	<p>
	VTS can use the following files to determine if the device target supports a
	specific HAL:
	</p>

	<ul>
	<li><code>/system/compatibility_matrix.xml</code>. Claims the HAL instances
	required by the framework. Example:
	<pre class="prettyprint">
	<hal format="hidl" optional="true">
	<name>android.hardware.vibrator</name>
	<version>1.0-1</version>
	<interface>
	<name>IVibrator</name>
	<instance>default</instance>
	</interface>
	</hal>
	</pre>
	<ul>
	<li>The <code>optional</code> attribute indicates if the HAL is strictly
	required by the framework.</li>
	<li>The file may contain multiple entries for the same HAL (with same name)
	but with different version and interfaces.</li>
	<li>The file may contain multiple <code>version</code> configurations for
	the same entry, indicating the framework can work with different versions.
	</li>
	<li><code>version1.0-1</code> means the framework can work with the lowest
	version 1.0, and does not require a version higher than 1.1.</li>
	</ul>
	</li>
	<li>Device <code>manifest.xml</code>. Claims the HAL instances provided by the
	vendor. Example:
	<pre class="prettyprint">
	<hal format="hidl">
	<name>android.hardware.vibrator</name>
	<transport>hwbinder</transport>
	<version>1.2</version>
	<interface>
	<name>IVibrator</name>
	<instance>default</instance>
	</interface>
	</hal>
	</pre>
	<ul>
	<li>The file may contain multiple entries for the same HAL (with same name)
	but with different version and interfaces.</li>
	<li>If the file contains only a single <code>version</code> configuration
	for an entry, <code>version1.2</code> means the vendor supports all versions
	from 1.0~1.2.</li>
	</ul>
	</li>
	<li><strong>lshal</strong>. A tool on device that shows runtime info about
	the HAL services registered with the <code>hwservicemanager</code>. Example:
	<pre class="prettyprint">
	android.hardware.vibrator@1.0::IVibrator/default
	</pre>
	<br><code>lshal</code> also shows all the HALs that with passthrough
	implementations (i.e having the corresponding <code>-impl.so</code> file on
	the device). Example:
	<pre class="prettyprint">
	android.hardware.nfc@1.0::I/ (/vendor/lib/hw/)
	android.hardware.nfc@1.0::I/ (/vendor/lib64/hw/)
	</pre>
	</li>
	</ul>

	<h2 id="compliance-tests">Compliance tests</h2>

	<p>
	For compliance tests, VTS relies on the vendor manifest to determine (and
	test) all HAL instances provided by the device. Decision flow:
	</p>

	<p>
	<img src="images/testability-check-compliance.png"
	alt="Testability check for compliance"
	title="Testability check for compliance">
	</p>
	<figcaption>
	<strong>Figure 1.</strong> Testability check for VTS compliance tests
	</figcaption>

	<h2 id="non-compliance-tests">Non-compliance tests</h2>

	<p>
	For non-compliance tests, VTS relies on the vendor manifest and
	<code>lshal</code> outputs to determine (and test) the experimental HALs not
	claimed in the <code>manifest.xml</code> file. Decision flow:
	</p>

	<p>
	<img src="images/testability-check-non-compliance.png"
	alt="Testability check for non-compliance"
	title="Testability check for non-compliance">
	<figcaption>
	<strong>Figure 2.</strong> Testability check for VTS non-compliance
	tests
	</figcaption>

	<h2 id="locating-the-vendor-manifest">Locating the vendor manifest</h2>

	<p>
	VTS checks for the vendor <code>manifest.xml</code> file in the following
	places in the following order:
	</p>

	<ol>
	<li><code>/vendor/etc/vintf/manifest.xml</code> + ODM manifest (If same HAL
	is defined in both places, ODM manifest overrides the one in
	<code>/vendor/etc/vintf/manifest.xml</code>)</li>
	<li><code>/vendor/etc/vintf/manifest.xml</code></li>
	<li>ODM <code>manifest.xml</code> file, loaded from the following files in
	the following order:
	<ol>
	<li><code>/odm/etc/vintf/manifest_$(ro.boot.product.hardware.sku).xml</code>
	</li>
	<li><code>/odm/etc/vintf/manifest.xml</code></li>
	<li><code>/odm/etc/manifest_$(ro.boot.product.hardware.sku).xml</code></li>
	<li><code>/odm/etc/manifest.xml</code></li>
	<li><code>/vendor/manifest.xml</code></li>
	</ol>
	</li>
	</ol>

	<h2 id="vts-testability-checker">VTS testability checker</h2>

	<p>
	The
	<code><a href="https://android.googlesource.com/platform/test/vts/+/master/utils/native/testability_checker/?q=vts_testability&g=0" class="external">
	vts_testibility_checker</a></code> is a binary packaged with VTS and used by
	VTS test framework at runtime to determine whether a given HAL test is
	testable or not. It is based on
	<code><a href="https://android.googlesource.com/platform/system/libvintf/+/master" class="external">libvintf</a></code>
	to load and parse the vendor manifest file and implements the decision flow
	described in the previous section.
	</p>

	<p>
	To use <code>vts_testability_check</code>:
	</p>

	<ul>
	<li>For a compliance test:
	<pre class="prettyprint">
	vts_testability_check -c -b <bitness> <hal@version>
	</pre>
	</li>
	<li>For a non-compliance test:
	<pre class="prettyprint">
	vts_testability_check -b <bitness> <hal@version>
	</pre>
	</li>
	</ul>

	<p>
	The output of <code>vts_testability_check</code> uses the following json
	format:
	</p>

	<pre class="prettyprint">
	{testable: <True/False> Instances: <list of instance names of HAL service>}
	</pre>

	<h2 id="determining-accessed-hals">Determining accessed HALs</h2>

	<p>
	To determine which HALs are accessed by VTS tests, ensure that each HAL test
	uses the
	<code><a href="https://android.googlesource.com/platform/test/vts/+/master/runners/target/vts_hal_hidl_target/VtsHalHidlTargetTestEnvBase.h" class="external">VtsHalHidlTargetTestEnvBase</a></code>
	template to register the HAL(s) accessed in the test. The VTS testing
	framework can then extract the registered HALs when pre-processing the test.
	</p>

	<p>
	For compliance tests, you can also check
	<code>/system/etc/vintf/manifest.xml</code>. If a HAL is defined here, VTS
	should test it. (For the HAL services provided by the system (e.g.
	<code>graphics.composer/vr</code>), the HALs are declared in
	<code>/system/manifest.xml</code>.)
	</p>

	</body>
	</html>