Automatic test retry

A test might fail for any reason, and sometimes simply re-running the test is enough to make it pass again (due to flakiness, such as from issues in the underlying infrastructure). You can configure Tradefed to conduct the retry automatically.

The core of auto-retry is to avoid re-running all of the tests; it re-runs only the failed tests, resulting in large savings in execution time.

Tradefed also supports running tests multiple times in order to detect flakiness via the iterations feature. In this case, all tests will be re-run, and the test will fail if any of the iterations fail.

Enable automatic retry

The automatic retry is controlled via the RetryDecision object which provides two options to enable the feature: max-testcase-run-count and retry-strategy.

max-testcase-run-count drives the number of retries or iterations that will be attempted. It sets an upper bound to avoid retrying forever. retry-strategy drives the decision of how to retry; see the following sections for more details.

Disable automatic retry

Use the following option:

--retry-strategy NO_RETRY

Retry failures

To retry test failures, use the following options:

--retry-strategy RETRY_ANY_FAILURE --max-testcase-run-count X

This will retry the failure until it passes or until the max number of retries is reached, whichever comes first.

Iterations

To re-run tests for a number of time, the following options can be used:

--retry-strategy ITERATIONS --max-testcase-run-count X

What do the results look like?

Result reporters by default will receive aggregated results of all attempts.

For example: a Fail and a Pass for RETRY_ANY_FAILURE will result in an aggregated Pass since the retry managed to clear the failure.

It is possible for reporters to receive the non-aggregated results. To do so, they need to extend the ISupportGranularResults interface that declares support for the granular (non-aggregated) results.

Implementation details

To enable auto-retry to rerun failures at the test case level, implement ITestFilterReceiver.

If you can't implement ITestFilterReceiver, you can implement IAutoRetriableTest to manually control how rerun operates. InstalledInstrumentationsTest is an example implementation of that interface.