Deqp test modules can be integrated to automated test systems in multiple ways. The best approach depends on the existing test infrastructure and target environment.
The primary output from a test run is always the test log file, i.e. the file
.qpa postfix. Full test results can be parsed from the test log. Console output is
debug information only and may not be available on all platforms.
Test binaries can be invoked directly from a test automation system. The test binary can be launched for a specific case, for a test set, or for all available tests. If a fatal error occurs during execution (such as certain API errors or a crash), the test execution will abort. For regression testing, the best approach is to invoke the test binaries for individual cases or small test sets separately, in order to have partial results available even in the event of hard failure.
The deqp comes with command line test execution tools that can be used in combination with the execution service to achieve a more robust integration. The executor detects test process termination and will resume test execution on the next available case. A single log file is produced from the full test session. This setup is ideal for lightweight test systems that don’t provide crash recovery facilities.
Command line test execution tools
The current command line tool set includes a remote test execution tool, a test log comparison generator for regression analysis, a test-log-to-CSV converter, a test-log-to-XML converter, and a testlog-to-JUnit converter.
The source code for these tools is in the
executor directory, and the binaries are built into the
Command line Test Executor
The command line Test Executor is a portable C++ tool for launching a test run
on a device and collecting the resulting logs from it over TCP/IP. The Executor
communicates with the execution service (execserver) on the target device.
Together they provide functionality such as recovery from test process crashes.
The following examples demonstrate how to use the command line Test Executor
--help for more details):
Example 1: Run GLES2 functional tests on an Android device:
executor --connect=127.0.0.1 --port=50016 --binaryname= com.drawelements.deqp/android.app.NativeActivity --caselistdir=caselists --testset=dEQP-GLES2.* --out=BatchResult.qpa --cmdline="--deqp-crashhandler=enable --deqp-watchdog=enable --deqp-gl-config-name=rgba8888d24s8"
Example 2: Continue a partial OpenGL ES 2 test run locally:
executor --start-server=execserver/execserver --port=50016 --binaryname=deqp-gles2 --workdir=modules/opengl --caselistdir=caselists --testset=dEQP-GLES2.* --exclude=dEQP-GLES2.performance.* --in=BatchResult.qpa --out=BatchResult.qpa
Test log CSV export and compare
The deqp has a tool for converting test logs (.
qpa files) into CSV files. The CSV output contains a list of test cases and their
results. The tool can also compare two or more batch results and list only the
test cases that have different status codes in the input batch results. The
comparison will also print the number of matching cases.
The output in CSV format is very practical for further processing with standard
command line utilities or with a spreadsheet editor. An additional, human-readable,
plain-text format can be selected using the following command line argument:
Example 1: Export test log in CSV format
testlog-to-csv --value=code BatchResult.qpa > Result_statuscodes.csv
testlog-to-csv --value=details BatchResult.qpa > Result_statusdetails.csv
Example 2: List differences of test results between two test logs
testlog-to-csv --mode=diff --format=text Device_v1.qpa Device_v2.qpa
Note: The argument
--value=code outputs the test result code, such as "Pass" or "Fail". The argument
--value=details selects the further explanation of the result or numerical value produced by a performance, capability, or accuracy test.
Test log XML export
Test log files can be converted to valid XML documents using the
testlog-to-xml utility. Two output modes are supported:
- Separate documents mode, where each test case and the
caselist.xmlsummary document are written to a destination directory
- Single file mode, where all results in the
.qpafile are written to single XML document.
Exported test log files can be viewed in a browser using an XML style sheet.
Sample style sheet documents (
testlog.css) are provided in the
doc/testlog-stylesheet directory. To render the log files in a browser, copy the two style sheet
files into the same directory where the exported XML documents are located.
If you are using Google Chrome, the files must be accessed over HTTP as Chrome
limits local file access for security reasons. The standard Python installation
includes a basic HTTP server that can be launched to serve the current
directory with the
python –m SimpleHTTPServer 8000 command. After launching the server, just point the Chrome browser to
http://localhost:8000 to view the test log.
Conversion to a JUnit test log
Many test automation systems can generate test run result reports from JUnit output. The deqp test log files can be converted to the JUnit output format using the testlog-to-junit tool.
The tool currently supports translating the test case verdict only. As JUnit supports only "pass" and "fail" results, a passing result of the deqp is mapped to "JUnit pass" and other results are considered failures. The original deqp result code is available in the JUnit output. Other data, such as log messages and result images, are not preserved in the conversion.