Atest is a command line tool that allows users to build, install and run Android tests locally. This markdown will explain how to use atest on the commandline to run android tests.
For instructions on writing tests go here. Importantly, when writing your test's build script file (Android.mk), make sure to include the variable LOCAL_COMPATIBILITY_SUITE. A good default to use for it is device-test.
Curious about how atest works? Want to add a feature but not sure where to begin? Go here. Just want to learn about the overall structure? Go here.
Before you can run atest, you first have to setup your environment.
From the root of the android source checkout, run the following command:
$ source build/envsetup.sh
Run the $ lunch command to bring up a “menu” of supported devices. Find the device that matches the device you have connected locally and run that command.
For instance, if you have an ARM device connected, you would run the following command:
$ lunch aosp_arm64-eng
This will set various environment variables necessary for running atest. It will also add the atest command to your $PATH.
Atest commands take the following form:
atest <optional arguments> <tests to run>
More information about -b, -i and -t can be found below under Specifying Steps: Build, Install or Run.
The positional argument <tests to run> should be a reference to one or more of the tests you'd like to run. Multiple tests can be run by separating test references with spaces.
Usage template: atest <reference_to_test_1> <reference_to_test_2>
Here are some simple examples:
atest FrameworksServicesTests
atest example/reboot
atest FrameworksServicesTests CtsJankDeviceTestCases
atest FrameworksServicesTests:ScreenDecorWindowTests
More information on how to reference a test can be found under Identifying Tests.
A <reference_to_test> can be satisfied by the test's Module Name, Module:Class, Class Name, TF Integration Test, File Path or Package Name.
Identifying a test by its module name will run the entire module. Input the name as it appears in the LOCAL_MODULE or LOCAL_PACKAGE_NAME variables in that test's Android.mk or Android.bp file.
Note: Use TF Integration Test to run non-module tests integrated directly into TradeFed.
Examples:
atest FrameworksServicesTests
atest CtsJankDeviceTestCases
Identifying a test by its class name will run just the tests in that class and not the whole module. Module:Class is the preferred way to run a single class. Module is the same as described above. Class is the name of the test class in the .java file. It can either be the fully qualified class name or just the basic name.
Examples:
atest FrameworksServicesTests:ScreenDecorWindowTests
atest FrameworksServicesTests:com.android.server.wm.ScreenDecorWindowTests
atest CtsJankDeviceTestCases:CtsDeviceJankUi
A single class can also be run by referencing the class name without the module name.
Examples:
atest ScreenDecorWindowTests
atest CtsDeviceJankUi
However, this will take more time than the equivalent Module:Class reference, so we suggest using a Module:Class reference whenever possible. Examples below are ordered by performance from the fastest to the slowest:
Examples:
atest FrameworksServicesTests:com.android.server.wm.ScreenDecorWindowTests
atest FrameworksServicesTests:ScreenDecorWindowTests
atest ScreenDecorWindowTests
To run tests that are integrated directly into TradeFed (non-modules), input the name as it appears in the output of the “tradefed.sh list configs” cmd.
Example:
atest example/reboot (runs this test)
atest native-benchmark (runs this test)
Both module-based tests and integration-based tests can be run by inputting the path to their test file or dir as appropriate. A single class can also be run by inputting the path to the class's java file. Both relative and absolute paths are supported.
Example - 2 ways to run the CtsJankDeviceTestCases module via path:
atest cts/tests/jankatest .Example - run a specific class within CtsJankDeviceTestCases module via path:
From android <repo root>: atest cts/tests/jank/src/android/jank/cts/ui/CtsDeviceJankUi.java
Example - run an integration test via path:
From android <repo root>: atest tools/tradefederation/contrib/res/config/example/reboot.xml
Atest supports searching tests from package name as well.
Examples:
atest com.android.server.wm
atest android.jank.cts
The -b, -i and -t options allow you to specify which steps you want to run. If none of those options are given, then all steps are run. If any of these options are provided then only the listed steps are run.
Note: -i alone is not currently support and can only be included with -t. Both -b and -t can be run alone.
atest -b <test>atest -t <test> atest -it <test> atest -bt <test> Atest now has the ability to force a test to skip its cleanup/teardown step. Many tests, e.g. CTS, cleanup the device after the test is run, so trying to rerun your test with -t will fail without having the --disable-teardown parameter. Use -d before -t to skip the test clean up step and test iteratively.
atest -d <test>
atest -t <test>
Note that -t disables both setup/install and teardown/cleanup of the device. So you can continue to rerun your test with just atest -t <test> as many times as you want.
It is possible to run only specific methods within a test class. While the whole module will still need to be built, this will greatly speed up the time needed to run the tests. To run only specific methods, identify the class in any of the ways supported for identifying a class (MODULE:CLASS, FILE PATH, etc) and then append the name of the method or method using the following template:
<reference_to_class>#<method1>
Multiple methods can be specified with commas:
<reference_to_class>#<method1>,<method2>,<method3>
Examples:
atest com.android.server.wm.ScreenDecorWindowTests#testMultipleDecors
atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChange,testRemoval
Here are the two preferred ways to run a single method, we're specifying the testFlagChange method. These are preferred over just the class name, because specifying the module or the java file location allows atest to find the test much faster:
atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChangeatest frameworks/base/services/tests/servicestests/src/com/android/server/wm/ScreenDecorWindowTests.java#testFlagChangeMultiple methods can be ran from different classes and modules:
atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChange,testRemoval ScreenDecorWindowTests#testMultipleDecors
To run multiple classes, deliminate them with spaces just like you would when running multiple tests. Atest will handle building and running classes in the most efficient way possible, so specifying a subset of classes in a module will improve performance over running the whole module.
Examples:
two classes in same module:
atest FrameworksServicesTests:ScreenDecorWindowTests FrameworksServicesTests:DimmerTests
two classes, different modules:
atest FrameworksServicesTests:ScreenDecorWindowTests CtsJankDeviceTestCases:CtsDeviceJankUi
Generate pre-patch or post-patch metrics without running regression detection:
Examples:
atest <test> --generate-baseline <optional iter>
atest <test> --generate-new-metrics <optional iter>
Local regression detection can be run in three options:
Provide a folder containing baseline (pre-patch) metrics (generated previously). Atest will run the tests n (default 5) iterations, generate a new set of post-patch metrics, and compare those against existing metrics.
Example:
atest <test> --detect-regression </path/to/baseline> --generate-new-metrics <optional iter>
Provide a folder containing post-patch metrics (generated previously). Atest will run the tests n (default 5) iterations, generate a new set of pre-patch metrics, and compare those against those provided. Note: the developer needs to revert the device/tests to pre-patch state to generate baseline metrics.
Example:
atest <test> --detect-regression </path/to/new> --generate-baseline <optional iter>
Provide 2 folders containing both pre-patch and post-patch metrics. Atest will run no tests but the regression detection algorithm.
Example:
atest --detect-regression </path/to/baseline> </path/to/new>