There are two suites of tests in the Android Runtime (ART):
ART run-tests are tests exercising the runtime using Dex bytecode. They are written in Java and/or Smali (compiled/assembled as Dex bytecode) and sometimes native code (written as C/C++ testing libraries). Some tests also make use of the Jasmin assembler or the ASM bytecode manipulation tool. Run-tests are executed on the ART runtime (
dalvikvm), possibly preceded by a pre-optimization of the Dex code (using
The run-tests are identified by directories in this
test directory, named with a numeric prefix and containing an
info.txt file. For most run tests, the sources are in the
src subdirectory. Sources found in the
src2 directory are compiled separately but to the same output directory; this can be used to exercise “API mismatch” situations by replacing class files created in the first pass. The
src-ex directory is built separately, and is intended for exercising class loaders. Resources can be stored in the
res directory, which is distributed together with the executable files.
The run-tests logic lives in the
test/run-test Bash script. The execution of a run-test has three main parts: building the test, running the test, and checking the test's output. By default, these three steps are implemented by three Bash scripts located in the
test/etc directory (
default-check). These scripts rely on environment variables set by
The default logic for all of these these steps (build, run, check) is overridden if the test's directory contains a Bash script named after the step (i.e.
check). Note that the default logic of the “run” step is actually implemented in the “JAR runner” (
test/etc/run-test-jar), invoked by
After the execution of a run-test, the check step‘s default behavior (implemented in
test/etc/default-check) is to respectively compare its standard output and standard error with the contents of the
expected-stderr.txt files contained in the test’s directory; any mismatch triggers a test failure.
test/run-test script handles the execution of a single run-test in a given configuration. The Python script
test/testrunner/testrunner.py is a convenient script handling the construction and execution of multiple tests in one configuration or more.
To see the invocation options supported by
testrunner.py, run these commands from the Android source top-level directory:
Some ART run-tests, known as “Checker tests”, perform additional checks on ART's compiler. They are identified by their name, which match the
^[0-9]+-checker-.* regular expression (e.g.
Checker assertions are annotations in a run-test‘s (Java and Smali) source files verifying the behavior of the ART compiler when compiling the corresponding Dex code. They are checked by the
checker tool (see directory
art/tools/checker) against a c1visualizer-style (
.cfg) file emitted by
dex2oat, containing control-flow graphs (CFGs) for compiled methods at each step (pass) in the compiler’s pipeline, as well as the emitted assembly code.
ART gtests are written in C++ using the Google Test framework. These tests exercise various aspects of the runtime (the logic in
libart-compiler, etc.) and its binaries (
oatdump, etc.). Some of them are used as unit tests to verify a particular construct in ART. These tests may depend on some test Dex files and core images.
ART gtests are defined in various directories within the ART project (usually in the same directory as the code they exercise). Their source files usually end with the suffix
_test.cc. The construction logic of these tests is implemented in ART's build system (
Android*.mk files). On host, these gtests can be run by executing
m test-art-host-gtest. On device, the recommended approach is to run these tests in a chroot environment (see
README.chroot.md in this directory).
All tests in either suite can be run using the
art/test.py script. Additionally, run-tests can be run individually. All of the tests can be run on the build host, on a USB-attached device, or using the build host “reference implementation”.
ART also supports running target (device) tests in a chroot environment (see
README.chroot.md in this directory). This is currently the recommended way to run tests on target (rather than using
To see command flags run:
$ art/test.py -h
In general all tests require some dependencies to be built before they can be run. In general you can pass the
--build-dependencies flag (also available as short option -b) to
art/test.py program to automatically build required dependencies. One can also directly use the various
test-art-...-dependencies targets listed below.
$ # Build test files $ m test-art-host-run-test-dependencies $ # Run the tests $ art/test.py --host
$ art/test.py -b --host
$ # Build test files $ m test-art-target-run-test-dependencies $ # Run the tests $ art/test.py --target
$ art/test.py -b --target
$ art/test.py --host -g
$ art/test.py --target -g
$ # Build test files $ m test-art-host-run-test-dependencies $ art/test.py --host -r
$ art/test.py -b --host -r
$ art/test.py --target -r
$ # Build test files $ m test-art-host-run-test-dependencies $ # Run the tests $ art/test.py --host -r -t 001-HelloWorld
$ art/test.py -b --host -r -t 001-HelloWorld
$ art/test.py --target -r -t 001-HelloWorld
$ find out/host/ -type f -name art_runtime_tests # Find the path of the test. $ out/host/linux-x86/nativetest/art_runtime_tests/art_runtime_tests
Add “--no_isolate” to run the tests one by one in single process (disable forking). Add “--gtest_filter=...” to select specific sub-test(s) to run. Prefix by "gdb --args " to run the test in gdb.
Both ART run-tests and gtests are run continuously as part of ART's continuous integration. In addition, two other test suites are run continuously on this service: Libcore tests and JDWP tests.