Use Suite Retry

A suite tends to include several test modules and can reach quite a large test corpus size. For example, the Android Compatibility Test Suite (CTS) includes hundreds of modules and hundreds of thousands test cases.

It becomes possible for a large amount of tests to fail due to poor isolation or devices going into a bad state.

The suite retry feature is meant to address those cases: It allows you to retry the failures only instead of the full suites in order to rule out flakiness and poor isolation. If a test is consistently failing, the retry will also fail; and you get a much stronger signal that there is a real issue.

Implement suite retry

The retry of results involves reading the previous results and re-running the previous invocation.

The main interface driving the retry is ITestSuiteResultLoader, which allows you to load a previous result, and the previous command line.

The RetryRescheduler then uses this information to recreate the previous command and populate some filters in order to re-run only the previous failures or not executed tests.

Example suite retry: CTS

The retry configuration in CTS is:

<configuration description="Runs a retry of a previous CTS session.">
    <object type="previous_loader" class="com.android.compatibility.common.tradefed.result.suite.PreviousResultLoader" />
    <test class="com.android.tradefed.testtype.suite.retry.RetryRescheduler" />

    <logger class="com.android.tradefed.log.FileLogger">
        <option name="log-level-display" value="WARN" />
    </logger>
</configuration>

This is applicable to most of the suites that extend it, for example VTS).

It would be invoked via:

cts-tradefed run retry --retry <session>

The session would be found by listing the previous results in the CTS console:

cts-tf > l r
Session  Pass  Fail  Modules Complete  Result Directory     Test Plan  Device serial(s)  Build ID   Product
0        2092  30    148 of 999        2018.10.29_14.12.57  cts        [serial]          P          Pixel

The exact original command will be reloaded and re-run with extra filters. This means that if your original command included some options, they would also be part of the retry.

For example:

cts-tradefed run cts-dev -m CtsGestureTestCases

The retry of the above would always be bounded to CtsGestureTestCases since we are retrying a command that involved only it.

Configure retry for CTS-style suite

In order for the retry to work, the previous results need to be exported in proto format. The following needs to be added:

<result_reporter class="com.android.compatibility.common.tradefed.result.suite.CompatibilityProtoResultReporter" />

This needs to be added to the XML configuration of the main command, and it will result in a test-record.pb file to be created in the result folder.

The CTS retry then loads data from a combination of the test-record.pb and the existing test_result.xml to prepare the retry invocation.