src/compatibility/cts/run.jd

page.title=Running CTS tests
@jd:body

<!--
    Copyright 2015 The Android Open Source Project

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-->
<div id="qv-wrapper">
  <div id="qv">
    <h2>In this document</h2>
    <ol id="auto-toc">
    </ol>
  </div>
</div>

<h2 id=cts_tradefed>Using the CTS tradefed</h2>
<p>See the <a
href="{@docRoot}devices/tech/test_infra/tradefed/index.html">Trade Federation
Overview</a> for an explanation of the Trade Federation (tradefed or TF for
short) continuous test framework.</p>

<p>To run a test plan:</p>
<ol>
  <li>Connect at least one device.
  <li>Press the <strong>home</strong> button to set the device to the home screen at the start of CTS.
  <li>While a device is running tests, it must not be used for any other tasks
    and must be kept in a stationary position (to avoid triggering sensor activity)
    with the cameras pointing at an object that could be focused.
  <li>Do not press any keys on the device while the CTS is running. Pressing keys
    or touching the screen of a test device will interfere with the running tests
    and may lead to test failures.
  <li>Launch the CTS console by running the <em>cts-tradefed</em> script from
    the folder where the CTS package has been unzipped, e.g.
    <code>$ ./android-cts/tools/cts-tradefed</code>
  <li>You may start the default test plan (containing all of the test packages) by
    appending: <code>run cts --plan CTS</code> This will kick off all the CTS tests required for compatibility.
    Enter <code>list plans</code> to see a list of test plans in the repository.
    Enter <code>list packages</code> to see a list of test packages in the repository.
    See the CTS command
    reference or type help for a complete list of supported commands.
  <li>Alternately, you may run the CTS plan of your choosing from the command line
    using: <code>cts-tradefed run cts --plan
      <plan_name>
      </code>
<p class="note"><strong>Note:</strong> When running Android 6.0 (Marshmallow)
CTS only, we recommend you use the <code>--skip-preconditions</code> option to
skip the experimental pre-conditions feature that may cause issues for when
executing CTS tests.</p>
  <li>View test progress and results reported on the console.
  <li>If your device is Android 5.0 or later and declares support for an ARM and a
    x86 ABI, you should run both the ARM and x86 CTS packages.
</ol>
<h2 id=selecting_cts_plans>Selecting CTS plans</h2>
<p>The following test plans are available:</p>
<ul>
  <li><em>CTS</em>—all tests required for compatibility. </li>
  <li><em>Signature</em>—the signature verification of all public APIs </li>
  <li><em>Android</em>—tests for the Android APIs </li>
  <li><em>Java</em>—tests for the Java core library </li>
  <li><em>VM</em>—tests for ART or Dalvik </li>
  <li><em>Performance</em>—performance tests for your implementation </li>
</ul>
<p>These can be executed with the <code>run cts</code> command.</p>
<h2 id=cts_reference>CTS console command reference</h2>

<p class="table-caption" id="console-commands">
  <strong>Table 1.</strong> This table summarizes the CTS console commands for
various uses.</p>
<table>
  <tbody>
    <tr>
      <th>Host</th>
      <th>Description</th>
    </tr>
    <tr>
      <td><code>help</code></td>
      <td>Display a summary of the most commonly used commands</td>
    </tr>
    <tr>
      <td><code>help all</code></td>
      <td>Display the complete list of available commands</td>
    </tr>
    <tr>
      <td><code>exit</code></td>
      <td>Gracefully exit the CTS console. Console will close when all currently running tests are finished</td>
    </tr>
    <tr>
      <th>Run</th>
      <th>Description</th>
    </tr>
    <tr>
      <td><code>run cts</code></td>
      <td>Run the specified tests and displays progress information. One of
<code>--plan</code>, <code>--package</code>, <code>--class</code> or
<code>--continue-session</code> needs to be specified
        <p>The CTS console can accept other commands while tests are in progress </p>
        <p>If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests </p>
        <p>If more than one device is connected, the CTS host will choose a device automatically</p></td>
    </tr>
    <tr>
      <td><code>--plan &lt;test_plan_name&gt;</code></td>
      <td>Run the specified test plan</td>
    </tr>
    <tr>
      <td><code>-- package/-p &lt;test_package_name&gt;  [--package/-p &lt;test_package2&gt;...]</code></td>
      <td>Run the specified test packages</td>
    </tr>
    <tr>
      <td><code>--class/-c &lt;class_name&gt; [--method/-m &lt;test_method_name&gt;</code></td>
      <td>Run the specified test class and/or method</td>
    </tr>
    <tr>
      <td><code>--continue-session</code></td>
      <td>Run all not executed tests from previous CTS session; the sessions testResult.xml will be updated with the new results</td>
    </tr>
    <tr>
      <td><code>--shards &lt;number_of_shards&gt;</code></td>
      <td>Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel</td>
    </tr>
    <tr>
      <td><code>--serial/-s &lt;deviceID&gt;</code></td>
      <td>Run CTS on the specific device</td>
    </tr>
    <tr>
      <td><code>-t &lt;class_name&gt;#&lt;test_method_name&gt;</code></td>
      <td>Run a specific test method</td>
    </tr>
    <tr>
      <td><code>--force-abi 32|64</code></td>
      <td>On 64-bit devices, run the test against only the 32-bit or 64-bit ABI</td>
    </tr>
    <tr>
      <th>List</th>
      <th>Description</th>
    </tr>
    <tr>
      <td><code>list packages</code></td>
      <td>List all available test packages in the repository</td>
    </tr>
    <tr>
      <td><code>list plans</code></td>
      <td>List all available test plans in the repository</td>
    </tr>
    <tr>
      <td><code>list invocations</code></td>
      <td>List 'run' commands currently being executed on devices</td>
    </tr>
    <tr>
      <td><code>list commands</code></td>
      <td>List all 'run' commands currently in the queue waiting to be assigned to devices</td>
    </tr>
    <tr>
      <td><code>list results</code></td>
      <td>List CTS results currently stored in repository</td>
    </tr>
    <tr>
      <td><code>list devices</code></td>
      <td>List currently connected devices and their state
        <p> </p>
        <p>'Available' devices are functioning, idle devices, available for running tests</p>
        <p> </p>
        <p>'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests</p>
        <p> </p>
        <p>'Allocated' devices are devices currently running tests</td>
    </tr>
    <tr>
      <th>Add</th>
      <th>Description</th>
    </tr>
    <tr>
      <td><code>add derivedplan --plan &lt;plan_name&gt;<br>
        --result/-r<br>
        [pass | fail | timeout | notExecuted]<br>
        [--session/-s &lt;session_id&gt;]</code></td>
      <td>Create a plan derived from given result session; use this option to rerun reports and validate test issues</td>
    </tr>
  </tbody>
</table>