| page.title=Automating the tests |
| @jd:body |
| |
| <!-- |
| Copyright 2015 The Android Open Source Project |
| |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| |
| <div id="qv-wrapper"> |
| <div id="qv"> |
| <h2>In this document</h2> |
| <ol id="auto-toc"> |
| </ol> |
| </div> |
| </div> |
| |
| <h2 id=intro>Introduction</h2> |
| |
| <p>Deqp test modules can be integrated to automated test systems in multiple ways. |
| The best approach depends on the existing test infrastructure and target |
| environment.</p> |
| |
| <p>The primary output from a test run is always the test log file, i.e. the file |
| with a <code>.qpa</code> postfix. Full test results can be parsed from the test log. Console output is |
| debug information only and may not be available on all platforms.</p> |
| |
| <p>Test binaries can be invoked directly from a test automation system. The test |
| binary can be launched for a specific case, for a test set, or for all |
| available tests. If a fatal error occurs during execution (such as certain API |
| errors or a crash), the test execution will abort. For regression testing, the |
| best approach is to invoke the test binaries for individual cases or small test |
| sets separately, in order to have partial results available even in the event |
| of hard failure.</p> |
| |
| <p>The deqp comes with command line test execution tools that can be used in |
| combination with the execution service to achieve a more robust integration. |
| The executor detects test process termination and will resume test execution on |
| the next available case. A single log file is produced from the full test |
| session. This setup is ideal for lightweight test systems that don’t provide |
| crash recovery facilities.</p> |
| |
| <h2 id=command_line_test_execution_tools>Command line test execution tools</h2> |
| |
| <p>The current command line tool set includes a remote test execution tool, a test |
| log comparison generator for regression analysis, a test-log-to-CSV converter, |
| a test-log-to-XML converter, and a testlog-to-JUnit converter.</p> |
| |
| <p>The source code for these tools is in the <code>executor</code> directory, and the binaries are built into the <code><builddir>/executor</code> directory.</p> |
| |
| <h3 id=command_line_test_executor>Command line Test Executor</h3> |
| |
| <p>The command line Test Executor is a portable C++ tool for launching a test run |
| on a device and collecting the resulting logs from it over TCP/IP. The Executor |
| communicates with the execution service (execserver) on the target device. |
| Together they provide functionality such as recovery from test process crashes. |
| The following examples demonstrate how to use the command line Test Executor |
| (use <code>--help</code> for more details):</p> |
| |
| <h4 id=example_1_run_gles2_functional_tests>Example 1: Run GLES2 functional tests on an Android device:</h4> |
| |
| <pre> |
| executor --connect=127.0.0.1 --port=50016 --binaryname= |
| com.drawelements.deqp/android.app.NativeActivity |
| --caselistdir=caselists |
| --testset=dEQP-GLES2.* --out=BatchResult.qpa |
| --cmdline="--deqp-crashhandler=enable --deqp-watchdog=enable |
| --deqp-gl-config-name=rgba8888d24s8" |
| </pre> |
| |
| <h4 id=example_2_continue_a_partial_opengl>Example 2: Continue a partial OpenGL ES 2 test run locally:</h4> |
| |
| <pre> |
| executor --start-server=execserver/execserver --port=50016 |
| --binaryname=deqp-gles2 --workdir=modules/opengl |
| --caselistdir=caselists |
| --testset=dEQP-GLES2.* --exclude=dEQP-GLES2.performance.* --in=BatchResult.qpa |
| --out=BatchResult.qpa |
| </pre> |
| |
| <h3 id=test_log_csv_export_and_compare>Test log CSV export and compare</h3> |
| |
| <p>The deqp has a tool for converting test logs (.<code>qpa </code>files) into CSV files. The CSV output contains a list of test cases and their |
| results. The tool can also compare two or more batch results and list only the |
| test cases that have different status codes in the input batch results. The |
| comparison will also print the number of matching cases.</p> |
| |
| <p>The output in CSV format is very practical for further processing with standard |
| command line utilities or with a spreadsheet editor. An additional, human-readable, |
| plain-text format can be selected using the following command line argument: <code>--format=text</code></p> |
| |
| <h4 id=example_1_export_test_log_in_csv_format>Example 1: Export test log in CSV format</h4> |
| |
| <pre> |
| testlog-to-csv --value=code BatchResult.qpa > Result_statuscodes.csv |
| testlog-to-csv --value=details BatchResult.qpa > Result_statusdetails.csv |
| </pre> |
| |
| <h4 id=example_2_list_differences>Example 2: List differences of test results between two test logs</h4> |
| |
| <pre> |
| testlog-to-csv --mode=diff --format=text Device_v1.qpa Device_v2.qpa |
| </pre> |
| |
| <p class="note"><strong>Note:</strong> The argument <code>--value=code</code> outputs the test result code, such as "Pass" or "Fail". The argument <code>--value=details</code> selects the further explanation of the result or numerical value produced by a performance, capability, or accuracy test.</p> |
| |
| <h3 id=test_log_xml_export>Test log XML export</h3> |
| |
| <p>Test log files can be converted to valid XML documents using the <code>testlog-to-xml</code> utility. Two output modes are supported: </p> |
| |
| <ul> |
| <li> Separate documents mode, where each test case and the <code>caselist.xml</code> summary document are written to a destination directory |
| <li> Single file mode, where all results in the <code>.qpa</code> file are written to single XML document. |
| </ul> |
| |
| <p>Exported test log files can be viewed in a browser using an XML style sheet. |
| Sample style sheet documents (<code>testlog.xsl</code> and <code>testlog.css</code>) are provided in the <code>doc/testlog-stylesheet</code> directory. To render the log files in a browser, copy the two style sheet |
| files into the same directory where the exported XML documents are located.</p> |
| |
| <p>If you are using Google Chrome, the files must be accessed over HTTP as Chrome |
| limits local file access for security reasons. The standard Python installation |
| includes a basic HTTP server that can be launched to serve the current |
| directory with the <code>python –m SimpleHTTPServer 8000</code> command. After launching the server, just point the Chrome browser to <code>http://localhost:8000</code> to view the test log.</p> |
| |
| <h3 id=conversion_to_a_junit_test_log>Conversion to a JUnit test log</h3> |
| |
| <p>Many test automation systems can generate test run result reports from JUnit |
| output. The deqp test log files can be converted to the JUnit output format |
| using the testlog-to-junit tool. </p> |
| |
| <p>The tool currently supports translating the test case verdict only. As JUnit |
| supports only "pass" and "fail" results, a passing result of the deqp is mapped |
| to "JUnit pass" and other results are considered failures. The original deqp |
| result code is available in the JUnit output. Other data, such as log messages |
| and result images, are not preserved in the conversion.</p> |