en/compatibility/cts/development.html - platform/docs/source.android.com - Git at Google

 page.title=CTS Development
 @jd:body

 <!--
     Copyright 2015 The Android Open Source Project

     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.
     You may obtain a copy of the License at

         http://www.apache.org/licenses/LICENSE-2.0

     Unless required by applicable law or agreed to in writing, software
     distributed under the License is distributed on an "AS IS" BASIS,
     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
     See the License for the specific language governing permissions and
     limitations under the License.
 -->
 <div id="qv-wrapper">
   <div id="qv">
     <h2>In this document</h2>
     <ol id="auto-toc">
     </ol>
   </div>
 </div>

 <h2 id="initializing-your-repo-client">Initializing your Repo client</h2>
 <p>Follow the <a href="{@docRoot}source/downloading.html">instructions</a>
 to get and build the Android source code but specify a particular CTS branch
 name, for example<code>-b android-5.0_r2</code>, when issuing the <code>repo
 init</code> command. This assures your CTS changes will be included in the
 next CTS release and beyond.</p>

 <h2 id="building-and-running-cts">Building and running CTS</h2>

 <p>Execute the following commands to build CTS and start the interactive
 CTS console:</p>
 <p class="note"><strong>Note:</strong> You may supply one of these other values
 for <code>TARGET_PRODUCT</code> to build for different architectures:
 <code>aosp_x86_64</code> or <code>aosp_mips</code></p>
 <pre><code>cd <em>/path/to/android/root</em>
 make cts -j32 TARGET_PRODUCT=aosp_arm64
 cts-tradefed
 </code></pre>

 <p>At the cts-tf console, enter e.g.:</p>
 <pre><code>run cts --plan CTS
 </code></pre>

 <h2 id="writing-cts-tests">Writing CTS tests</h2>

 <p>CTS tests use JUnit and the Android testing APIs. Review the
 <a href="https://developer.android.com/tools/testing/testing_android.html">Testing and Instrumentation</a>
 tutorial while perusing the existing tests under the
 <code>cts/tests</code> directory. You will see that CTS tests mostly follow the same
 conventions used in other Android tests.</p>

 <p>Since CTS runs across many production devices, the tests must follow
 these rules:</p>
 <ul>
 <li>Must take into account varying screen sizes, orientations, and keyboard layouts.</li>
 <li>Only use public API methods. In other words, avoid all classes, methods, and fields that are annotated with the "hide" annotation.</li>
 <li>Avoid relying upon particular view layouts or depend on the dimensions of assets that may not be on some device.</li>
 <li>Don't rely upon root privileges.</li>
 </ul>

 <h3 id="test-naming-and-location">Test naming and location</h3>

 <p> Most CTS test cases target a specific class in the Android API. These tests
 have Java package names with a <code>cts</code> suffix and class names with the
 <code>Test</code> suffix. Each test case consists of multiple tests, where each
 test usually exercises a particular method of the class being tested.
 These tests are arranged in a directory structure where tests are grouped into
 different categories such as "widgets" and "views." </p>

 <p>
 For example, the CTS test for the Java package
 <code>android.widget.TextView</code> is
 <code>android.widget.cts.TextViewTest</code> with its Java package name as
 <code>android.widget.cts</code> and its class name as
 <code>TextViewTest</code>.  </p>

 <ul>
 <li><strong>Java package name</strong><br/>The Java package name for the CTS tests
 is the package name of the class that the test is testing, followed by ".cts".
 For our example, the package name would be <code>android.widget.cts</code>.

 <li><strong>Class name</strong><br/>The class name for CTS tests is <strong>
 </strong>name of the class that the test is testing with "Test" appended. (For
 example,  if a test is targeting <code>TextView</code>, the class name should be
 <code>TextViewTest</code>.)

 <li><strong>Module name (CTS v2 only)</strong><br/>CTS v2 organizes tests by module.
  The module name is usually the second string of the Java package name (in our
 example, <code>widget</code>), although it does not have to be.</li>
 </ul>

 <p>
 The directory structure and sample code depend on whether you are using CTS v1
 or CTS v2.
 </p>
 <h4 id="cts-v1">CTS v1</h4>
 <p>
 For Android 6.0 and earlier, use CTS v1.  For CTS v1, the sample code is at
 <code>cts/tests/tests/example</code>.
 </p>
 <p>
 The directory structure in CTS v1 tests looks like this:
 </p>
 <pre class="no-pretty-print">
 cts/
   tests/
     tests/
       <em>package-name</em>/
         Android.mk
         AndroidManifest.xml
         src/
           android/
             <em>package-name</em>/
               SampleDeviceActivity.java
               cts/
                 SampleDeviceTest.java
 </pre>
 <h4 id="cts-v2"><strong>CTS v2</strong></h4>
 <p>
 For Android 7.0 and later, use CTS v2.  For details, see <a
 href="https://android.googlesource.com/platform/cts/+/master/tests/sample/">the
 sample test in AOSP</a>.
 </p>
 <p>
 The CTS v2 directory structure looks like this:
 </p>

 <pre class="no-pretty-print">
 cts/
   tests/
     <em>module-name</em>/
       Android.mk
       AndroidManifest.xml
       src/
         android/
           <em>package-name</em>/
             SampleDeviceActivity.java
             cts/
               SampleDeviceTest.java
 </pre>

 <h3 id="new-sample-packages">New sample packages</h3>

 <p>When adding new tests, there may not be an existing directory to place your
 test.  In those cases, you'll need to create the directory and copy the
 appropriate sample files.</p>

 <h4 id="cts-v1">CTS v1</h4>

 <p> If you are using CTS v1, refer to the example under
 <code>cts/tests/tests/example</code> and create a new directory. Also,
 make sure to add your new package's module name from its <code>Android.mk</code>
 to <code>CTS_COVERAGE_TEST_CASE_LIST</code>  in
 <code>cts/CtsTestCaseList.mk</code>. This Makefile is used by
 <code>build/core/tasks/cts.mk</code> to combine all the tests together to create
 the final CTS package.  </p>

 <h4 id="cts-v2">CTS v2</h4>
 <p>
 Use the sample test
 <code><a href="https://android.googlesource.com/platform/cts/+/master/tests/sample/">/cts/tests/sample/</a></code>
 to quick start your new test module with following steps:
 </p>

 <ol>
   <li>Run this command to create the test directory and copy sample files to it:
     <pre class="no-pretty-print">$ mkdir cts/tests/<i>module-name</i> && cp -r cts/tests/sample/* cts/tests/<i>module-name</i></pre>
   <li>Navigate to <code>cts/tests/<em>module-name</em></code> and substitute all instances of
 "[Ss]ample" following the recommended naming convention from above.
   <li>Update <code>SampleDeviceActivity</code> to exercise the feature you're testing.
   <li>Update <code>SampleDeviceTest</code> to ensure the activity succeeds or logs its
 errors.</li>
 </ol>

 <h4><strong>Additional directories</strong></h4>

 <p> Other Android directories such as <code>assets</code>, <code>jni</code>,
 <code>libs</code> and <code>res</code> can also be added. To add JNI code,
 create a directory in the root of the project next to <code>src</code> with the native
 code and an <code>Android.mk</code> in it.</p>

 <p>
 The makefile typically contains the following settings:
 </p>
 <pre>
 LOCAL_PATH := $(call my-dir)
 include $(CLEAR_VARS)
 LOCAL_MODULE := libCtsSample_jni

 # don't include this package in any target
 LOCAL_MODULE_TAGS := optional
 LOCAL_SRC_FILES := <i>list of source code files</i>
 LOCAL_C_INCLUDES := $(JNI_H_INCLUDE)

 # Tag this module as a cts test artifact
 LOCAL_COMPATIBILITY_SUITE := cts
 LOCAL_SHARED_LIBRARIES := libnativehelper
 LOCAL_SDK_VERSION := current
 include $(BUILD_SHARED_LIBRARY)
 </pre>

 <h4>Android.mk file</h4>

 <p> Finally, the <code>Android.mk</code> file in the root of the project will
 need to be modified to build the native code and depend on it, as shown
 below:</p>

 <pre>
 # All tests should include android.test.runner.
 LOCAL_JAVA_LIBRARIES := android.test.runner

 # Includes the jni code as a shared library
 LOCAL_JNI_SHARED_LIBRARIES := libCtsSample_jni

 # Include for InstrumentationCtsTestRunner
 LOCAL_STATIC_JAVA_LIBRARIES := ctstestrunner...
 LOCAL_SDK_VERSION := currentinclude $(BUILD_CTS_PACKAGE)

 #Tells make to look in subdirectories for more make files to include
 include $(call all-makefiles-under,$(LOCAL_PATH))
 </pre>

 <h2 id="Fix-remove-tests">Fix or remove tests</h2>
 <p>Besides adding new tests there are other ways to contribute to CTS: Fix or
 remove tests annotated with "BrokenTest" or "KnownFailure."</p>

 <h2 id="submitting-your-changes">Submitting your changes</h2>
 <p>Follow the <a href="{@docRoot}source/submit-patches.html">Submitting Patches workflow</a>
 to contribute changes to CTS. A reviewer
 will be assigned to your change, and your change should be reviewed shortly!</p>

 <h2 id="release-schedule">Release schedule and branch information</h2>

 <p>CTS releases follow this schedule.</p>

 <p class="note"><strong>Note</strong>: This schedule is tentative and may be
 updated from time to time as CTS for the given Android version matures.</p>

 <table>
 <tr>
   <th>Version</th>
   <th>Branch</th>
   <th>Frequency</th>
 </tr>
 </thead>
 <tbody>
 <tr>
   <td>7.0</td>
     <td>nougat-cts-dev</td>
       <td>Monthly</td>
       </tr>
 <tr>
   <td>6.0</td>
   <td>marshmallow-cts-dev</td>
   <td>Monthly</td>
 </tr>
 <tr>
   <td>5.1</td>
   <td>lollipop-mr1-cts-dev</td>
   <td>Monthly</td>
 </tr>
 <tr>
   <td>5.0</td>
   <td>lollipop-cts-dev</td>
   <td>No releases planned</td>
 </tr>
 <tr>
   <td>4.4</td>
   <td>kitkat-cts-dev</td>
   <td>No releases planned</td>
 </tr>
 <tr>
   <td>4.3</td>
   <td>jb-mr2-cts-dev</td>
   <td>No releases planned</td>
 </tr>
 <tr>
   <td>4.2</td>
   <td>jb-mr1.1-cts-dev</td>
   <td>No releases planned</td>
 </tr>
 </table>

 <h3 id="important-dates">Important Dates during month of the release</h3>

 <ul>
   <li><strong>End of 1st Week</strong>: Code Freeze.  At this point,
 submissions on the current branch will no longer be accepted and will not be
 included in the next version of CTS.  Once we have chosen a candidate for
 release, the branch will again be open and accepting new submissions.

   <li><strong>Second or third week</strong>: CTS is published in the Android
 Open Source Project (AOSP).
 </ul>

 <h3 id="auto-merge">Auto-merge flow</h3>

 <p>CTS development branches have been set up so that changes submitted to each
 branch will automatically merge as below:<br>
 jb-dev-> jb-mr1.1-cts-dev -> jb-mr2-cts-dev  -> kitkat-cts-dev ->
 lollipop-cts-dev -> lollipop-mr1-cts-dev -> marshmallow-cts-dev ->
 nougat-cts-dev -> &lt;private-development-branch for Android N MR1&gt;</p>

 <p>If a changelist (CL) fails to merge correctly, the author of the CL will get
 an email with instructions on how to resolve the conflict. In most of the
 cases, the author of the CL can use the instructions to skip the auto-merge of
 the conflicting CL.</p>
	page.title=CTS Development
	@jd:body

	<!--
	Copyright 2015 The Android Open Source Project

	Licensed under the Apache License, Version 2.0 (the "License");
	you may not use this file except in compliance with the License.
	You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing, software
	distributed under the License is distributed on an "AS IS" BASIS,
	WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	See the License for the specific language governing permissions and
	limitations under the License.
	-->
	<div id="qv-wrapper">
	<div id="qv">
	<h2>In this document</h2>
	<ol id="auto-toc">
	</ol>
	</div>
	</div>

	<h2 id="initializing-your-repo-client">Initializing your Repo client</h2>
	<p>Follow the <a href="{@docRoot}source/downloading.html">instructions</a>
	to get and build the Android source code but specify a particular CTS branch
	name, for example<code>-b android-5.0_r2</code>, when issuing the <code>repo
	init</code> command. This assures your CTS changes will be included in the
	next CTS release and beyond.</p>

	<h2 id="building-and-running-cts">Building and running CTS</h2>

	<p>Execute the following commands to build CTS and start the interactive
	CTS console:</p>
	<p class="note"><strong>Note:</strong> You may supply one of these other values
	for <code>TARGET_PRODUCT</code> to build for different architectures:
	<code>aosp_x86_64</code> or <code>aosp_mips</code></p>
	<pre><code>cd <em>/path/to/android/root</em>
	make cts -j32 TARGET_PRODUCT=aosp_arm64
	cts-tradefed
	</code></pre>

	<p>At the cts-tf console, enter e.g.:</p>
	<pre><code>run cts --plan CTS
	</code></pre>

	<h2 id="writing-cts-tests">Writing CTS tests</h2>

	<p>CTS tests use JUnit and the Android testing APIs. Review the
	<a href="https://developer.android.com/tools/testing/testing_android.html">Testing and Instrumentation</a>
	tutorial while perusing the existing tests under the
	<code>cts/tests</code> directory. You will see that CTS tests mostly follow the same
	conventions used in other Android tests.</p>

	<p>Since CTS runs across many production devices, the tests must follow
	these rules:</p>
	<ul>
	<li>Must take into account varying screen sizes, orientations, and keyboard layouts.</li>
	<li>Only use public API methods. In other words, avoid all classes, methods, and fields that are annotated with the "hide" annotation.</li>
	<li>Avoid relying upon particular view layouts or depend on the dimensions of assets that may not be on some device.</li>
	<li>Don't rely upon root privileges.</li>
	</ul>

	<h3 id="test-naming-and-location">Test naming and location</h3>

	<p> Most CTS test cases target a specific class in the Android API. These tests
	have Java package names with a <code>cts</code> suffix and class names with the
	<code>Test</code> suffix. Each test case consists of multiple tests, where each
	test usually exercises a particular method of the class being tested.
	These tests are arranged in a directory structure where tests are grouped into
	different categories such as "widgets" and "views." </p>

	<p>
	For example, the CTS test for the Java package
	<code>android.widget.TextView</code> is
	<code>android.widget.cts.TextViewTest</code> with its Java package name as
	<code>android.widget.cts</code> and its class name as
	<code>TextViewTest</code>. </p>

	<ul>
	<li><strong>Java package name</strong><br/>The Java package name for the CTS tests
	is the package name of the class that the test is testing, followed by ".cts".
	For our example, the package name would be <code>android.widget.cts</code>.

	<li><strong>Class name</strong><br/>The class name for CTS tests is <strong>
	</strong>name of the class that the test is testing with "Test" appended. (For
	example, if a test is targeting <code>TextView</code>, the class name should be
	<code>TextViewTest</code>.)

	<li><strong>Module name (CTS v2 only)</strong><br/>CTS v2 organizes tests by module.
	The module name is usually the second string of the Java package name (in our
	example, <code>widget</code>), although it does not have to be.</li>
	</ul>

	<p>
	The directory structure and sample code depend on whether you are using CTS v1
	or CTS v2.
	</p>
	<h4 id="cts-v1">CTS v1</h4>
	<p>
	For Android 6.0 and earlier, use CTS v1. For CTS v1, the sample code is at
	<code>cts/tests/tests/example</code>.
	</p>
	<p>
	The directory structure in CTS v1 tests looks like this:
	</p>
	<pre class="no-pretty-print">
	cts/
	tests/
	tests/
	<em>package-name</em>/
	Android.mk
	AndroidManifest.xml
	src/
	android/
	<em>package-name</em>/
	SampleDeviceActivity.java
	cts/
	SampleDeviceTest.java
	</pre>
	<h4 id="cts-v2"><strong>CTS v2</strong></h4>
	<p>
	For Android 7.0 and later, use CTS v2. For details, see <a
	href="https://android.googlesource.com/platform/cts/+/master/tests/sample/">the
	sample test in AOSP</a>.
	</p>
	<p>
	The CTS v2 directory structure looks like this:
	</p>

	<pre class="no-pretty-print">
	cts/
	tests/
	<em>module-name</em>/
	Android.mk
	AndroidManifest.xml
	src/
	android/
	<em>package-name</em>/
	SampleDeviceActivity.java
	cts/
	SampleDeviceTest.java
	</pre>

	<h3 id="new-sample-packages">New sample packages</h3>

	<p>When adding new tests, there may not be an existing directory to place your
	test. In those cases, you'll need to create the directory and copy the
	appropriate sample files.</p>

	<h4 id="cts-v1">CTS v1</h4>

	<p> If you are using CTS v1, refer to the example under
	<code>cts/tests/tests/example</code> and create a new directory. Also,
	make sure to add your new package's module name from its <code>Android.mk</code>
	to <code>CTS_COVERAGE_TEST_CASE_LIST</code> in
	<code>cts/CtsTestCaseList.mk</code>. This Makefile is used by
	<code>build/core/tasks/cts.mk</code> to combine all the tests together to create
	the final CTS package. </p>

	<h4 id="cts-v2">CTS v2</h4>
	<p>
	Use the sample test
	<code><a href="https://android.googlesource.com/platform/cts/+/master/tests/sample/">/cts/tests/sample/</a></code>
	to quick start your new test module with following steps:
	</p>

	<ol>
	<li>Run this command to create the test directory and copy sample files to it:
	<pre class="no-pretty-print">$ mkdir cts/tests/<i>module-name</i> && cp -r cts/tests/sample/* cts/tests/<i>module-name</i></pre>
	<li>Navigate to <code>cts/tests/<em>module-name</em></code> and substitute all instances of
	"[Ss]ample" following the recommended naming convention from above.
	<li>Update <code>SampleDeviceActivity</code> to exercise the feature you're testing.
	<li>Update <code>SampleDeviceTest</code> to ensure the activity succeeds or logs its
	errors.</li>
	</ol>

	<h4><strong>Additional directories</strong></h4>

	<p> Other Android directories such as <code>assets</code>, <code>jni</code>,
	<code>libs</code> and <code>res</code> can also be added. To add JNI code,
	create a directory in the root of the project next to <code>src</code> with the native
	code and an <code>Android.mk</code> in it.</p>

	<p>
	The makefile typically contains the following settings:
	</p>
	<pre>
	LOCAL_PATH := $(call my-dir)
	include $(CLEAR_VARS)
	LOCAL_MODULE := libCtsSample_jni

	# don't include this package in any target
	LOCAL_MODULE_TAGS := optional
	LOCAL_SRC_FILES := <i>list of source code files</i>
	LOCAL_C_INCLUDES := $(JNI_H_INCLUDE)

	# Tag this module as a cts test artifact
	LOCAL_COMPATIBILITY_SUITE := cts
	LOCAL_SHARED_LIBRARIES := libnativehelper
	LOCAL_SDK_VERSION := current
	include $(BUILD_SHARED_LIBRARY)
	</pre>

	<h4>Android.mk file</h4>

	<p> Finally, the <code>Android.mk</code> file in the root of the project will
	need to be modified to build the native code and depend on it, as shown
	below:</p>

	<pre>
	# All tests should include android.test.runner.
	LOCAL_JAVA_LIBRARIES := android.test.runner

	# Includes the jni code as a shared library
	LOCAL_JNI_SHARED_LIBRARIES := libCtsSample_jni

	# Include for InstrumentationCtsTestRunner
	LOCAL_STATIC_JAVA_LIBRARIES := ctstestrunner...
	LOCAL_SDK_VERSION := currentinclude $(BUILD_CTS_PACKAGE)

	#Tells make to look in subdirectories for more make files to include
	include $(call all-makefiles-under,$(LOCAL_PATH))
	</pre>

	<h2 id="Fix-remove-tests">Fix or remove tests</h2>
	<p>Besides adding new tests there are other ways to contribute to CTS: Fix or
	remove tests annotated with "BrokenTest" or "KnownFailure."</p>

	<h2 id="submitting-your-changes">Submitting your changes</h2>
	<p>Follow the <a href="{@docRoot}source/submit-patches.html">Submitting Patches workflow</a>
	to contribute changes to CTS. A reviewer
	will be assigned to your change, and your change should be reviewed shortly!</p>

	<h2 id="release-schedule">Release schedule and branch information</h2>

	<p>CTS releases follow this schedule.</p>

	<p class="note"><strong>Note</strong>: This schedule is tentative and may be
	updated from time to time as CTS for the given Android version matures.</p>

	<table>
	<tr>
	<th>Version</th>
	<th>Branch</th>
	<th>Frequency</th>
	</tr>
	</thead>
	<tbody>
	<tr>
	<td>7.0</td>
	<td>nougat-cts-dev</td>
	<td>Monthly</td>
	</tr>
	<tr>
	<td>6.0</td>
	<td>marshmallow-cts-dev</td>
	<td>Monthly</td>
	</tr>
	<tr>
	<td>5.1</td>
	<td>lollipop-mr1-cts-dev</td>
	<td>Monthly</td>
	</tr>
	<tr>
	<td>5.0</td>
	<td>lollipop-cts-dev</td>
	<td>No releases planned</td>
	</tr>
	<tr>
	<td>4.4</td>
	<td>kitkat-cts-dev</td>
	<td>No releases planned</td>
	</tr>
	<tr>
	<td>4.3</td>
	<td>jb-mr2-cts-dev</td>
	<td>No releases planned</td>
	</tr>
	<tr>
	<td>4.2</td>
	<td>jb-mr1.1-cts-dev</td>
	<td>No releases planned</td>
	</tr>
	</table>

	<h3 id="important-dates">Important Dates during month of the release</h3>

	<ul>
	<li><strong>End of 1st Week</strong>: Code Freeze. At this point,
	submissions on the current branch will no longer be accepted and will not be
	included in the next version of CTS. Once we have chosen a candidate for
	release, the branch will again be open and accepting new submissions.

	<li><strong>Second or third week</strong>: CTS is published in the Android
	Open Source Project (AOSP).
	</ul>

	<h3 id="auto-merge">Auto-merge flow</h3>

	<p>CTS development branches have been set up so that changes submitted to each
	branch will automatically merge as below:<br>
	jb-dev-> jb-mr1.1-cts-dev -> jb-mr2-cts-dev -> kitkat-cts-dev ->
	lollipop-cts-dev -> lollipop-mr1-cts-dev -> marshmallow-cts-dev ->
	nougat-cts-dev -> <private-development-branch for Android N MR1></p>

	<p>If a changelist (CL) fails to merge correctly, the author of the CL will get
	an email with instructions on how to resolve the conflict. In most of the
	cases, the author of the CL can use the instructions to skip the auto-merge of
	the conflicting CL.</p>