Trade Federation (Tradefed or TF for short) is a continuous test framework designed for running tests on Android devices. For example, Tradefed is used to run the Compatibility Test Suite (CTS) and Vendor Test Suite (VTS).
Trade Federation is a Java application which runs on a host computer, and communicates to one or more Android devices using ddmlib (the library behind DDMS) over adb.
We've listed some of TF's main features below, along with a couple sample usecases. That said, if you want to jump right in and get started, you can head straight for the Start Here page.
Features
- modular, flexible, scalable design
- has built in support for running many different types of Android tests: instrumentation, uiautomator, native/gtest, host-based JUnit, etc
- provides reliability and recovery mechanisms on top of adb
- supports scheduling and running tests on multiple devices in parallel
See Testing Through TF for the most up-to-date information about how to run your existing tests, such as Instrumentation.
Use cases
Trade Federation's modularity makes it straightforward to slot into environments with existing build, test, and reporting infrastructures. We list below a few demonstrative usecases where tradefed could enable efficient, scalable test practices.
First, it is useful to consider the landscape of potential usecases in terms of the question "which parts are modifiable, and what parts are static?" For instance, a Device OEM can modify the framework, the system, and the hardware, but has little or no influence over existing applications. An application developer, on the other hand, can modify the app, but has little control over most aspects of the system or the framework.
As a result, an entity in each usecase will have different testing goals, and will have different options in the case of a set of test failures. Despite these differences, Trade Federation can help make each of their test processes efficient, flexible, and scalable.
Device OEM
A Device OEM builds hardware, and will often tweak the Android system and frameworks to run well on that hardware. The OEM might strive to accomplish those goals while retaining stability and performance at the hardware and system levels, and making sure the framework changes don't break compatibility with existing applications.
The OEM could implement a device flashing module that will execute during the Target Setup stage of the lifecycle. That module would have complete control over the device during its execution period, which would allow it to potentially force the device into the bootloader, flash, and then force the device to reboot back into userspace mode. Combined with a module to tie into a continuous build system, this would allow the OEM to run tests on their device as they make changes to the system-level firmware and Java-level frameworks.
Once the device is fully booted, the OEM would be able to leverage existing JUnit-based tests, or write new ones, to verify the functionality of interest. Finally, they could write one or more result reporting modules to tie into existing test-result repositories, or to report results directly (for instance, by email).
App developer
An Application Developer builds an app which needs to run well across a variety of platform versions and a variety of devices. If an issue comes up on a particular platform version and/or device, the only remedy is to add a workaround and move on. For larger developers, the test process might be incorporated into a continuous build sequence. For smaller developers, it might be kicked off periodically or by hand.
Most app developers would use the apk test installation modules that already exist in TF. There's a version that installs from the local filesystem, as well as a version that can install apks downloaded from a build service. It is important to note that the latter version would continue to work properly with arbitrarily many TF instances running on the same host machine.
Because of TF's proficiency at dealing with multiple devices, it would be straightforward to classify each test result by the type of device that was used for that test. Thus, TF could potentially generate a 2-dimensional (or multi-dimensional) compatibility matrix for every build of the application.
Testing service
A Test Service might, for instance, allow app developers to submit apps and run tests on devices instrumented with power-measurement tools to determine power usage for the app. This differs from the prior two usecases in that the service builder does not control the devices or the applications that are being run.
Because Trade Federation can run any Java class that implements the simple
IRemoteTest interface, it's
trivial to write drivers that can coordinate some external piece of hardware with the test case
that's being run on the device. The driver itself can spawn Threads, send requests to other
servers, or do anything else that it might need. Moreover, the simplicity and versatility of the
result reporting interface,
ITestInvocationListener, means that it is likewise straightforward to
represent arbitrary test results (including, for instance, numerical power metrics) into the
standard result reporting pipeline.