A suite tends to include several test modules and can reach quite a large test corpus size. For example, the Android Compatibility Test Suite (CTS) includes hundreds of modules and hundreds of thousands test cases.
It becomes possible for a large amount of tests to fail due to poor isolation or devices going into a bad state.
The suite retry feature is meant to address those cases: It allows you to retry the failures only instead of the full suites in order to rule out flakiness and poor isolation. If a test is consistently failing, the retry will also fail; and you get a much stronger signal that there is a real issue.
Implement suite retry
The retry of results involves reading the previous results and re-running the previous invocation.
The main interface driving the retry is ITestSuiteResultLoader, which allows you to load a previous result, and the previous command line.
The RetryRescheduler then uses this information to recreate the previous command and populate some filters in order to re-run only the previous failures or not executed tests.
Example suite retry: CTS
The retry configuration in CTS is:
<configuration description="Runs a retry of a previous CTS session."> <object type="previous_loader" class="com.android.compatibility.common.tradefed.result.suite.PreviousResultLoader" /> <test class="com.android.tradefed.testtype.suite.retry.RetryRescheduler" /> <logger class="com.android.tradefed.log.FileLogger"> <option name="log-level-display" value="WARN" /> </logger> </configuration>
This is applicable to most of the suites that extend it, for example VTS).
It would be invoked via:
cts-tradefed run retry --retry <session>
The session would be found by listing the previous results in the CTS console:
cts-tf > l r Session Pass Fail Modules Complete Result Directory Test Plan Device serial(s) Build ID Product 0 2092 30 148 of 999 2018.10.29_14.12.57 cts [serial] P Pixel
The exact original command will be reloaded and re-run with extra filters. This means that if your original command included some options, they would also be part of the retry.
cts-tradefed run cts-dev -m CtsGestureTestCases
The retry of the above would always be bounded to
we are retrying a command that involved only it.
Configure retry for CTS-style suite
In order for the retry to work, the previous results need to be exported in proto format. The following needs to be added:
<result_reporter class="com.android.compatibility.common.tradefed.result.suite.CompatibilityProtoResultReporter" />
This needs to be added to the XML configuration of the main command, and it will
result in a
test-record.pb file to be created in the result folder.
The CTS retry then loads data from a combination of the
test_result.xml to prepare the retry invocation.