Testing Device Provisioning

For devices running Android 6 or Android 7, you can test device provisioning using the Android Enterprise (AE) Test Harness, which is a test suite for validating the enterprise compatibility of Android devices. The harness includes support apps, test cases, configuration files, and a test runner (afw-test-tradefed) built on cts-tradefed. Before setting up the AE Test Harness, be sure to complete Provisioning for Device Administration.

For devices running Android 8 or higher, use of the AE Test Harness is deprecated.

Setting up a development environment

The development environment for the AE Test Harness is similar to Android OS. Follow the steps in Requirements to set up a development machine.

Downloading source code

Download the AE Test Harness source code using the steps in Downloading the Source. The AE Test Harness source code is in the ./test/AfwTestHarness project. The branch name determines the version of AE Test Harness to download (each Android platform has a separate version of AE Test Harness). For example, the branch name for Android 7.0 Nougat is afw-test-harness-nougat-dev. To initialize the repo and download source code for this branch, use the following commands:

mkdir WORKING_DIRECTORY
cd WORKING_DIRECTORY
git config --global user.name "Your Name"
git config --global user.email "you@example.com"
repo init -u https://android.googlesource.com/platform/manifest -b afw-test-harness-nougat-dev
repo sync -j24

To check out the source code for a different version, specify the branch with the corresponding tag. Available branches include:

Branch Name	Supported Android Platform
afw-test-harness-nougat-dev	Android 7.0
afw-test-harness-2.1	Android 7.0
afw-test-harness-marshmallow-dev	Android 6.0
afw-test-harness-1.5	Android 6.0

Other dependency projects required to build the harness are also downloaded with the source code.

Viewing in Android Studio

To view and edit the source code in Android Studio:

Run the following commands

make idegen
development/tools/idegen/idegen.sh

In Android Studio, open android.ipr.

The AE Test Harness source code is in test/AfwTestHarness.

Configuring the AE Test Harness

You can customize the harness by configuring test/AfwTestHarness/afw-test.props. To run the harness successfully, complete the following steps:

Configure the Wi-Fi network in afw-test.props with the following properties:

wifi_ssid
wifi_password (optional)
wifi_security_type (optional, available options are: NONE, WEP or WPA)

Obtain at least one account from a domain that is bound to Test DPC as its device policy controller. Specify the details in afw-test.props with the following properties:
```
work_account_username
work_account_password
```
The AE Test Harness uses Test DPC to test provisioning flows, so accounts must bind to Test DPC to run the test harness.

Building the AE Test Harness

Initialize the build configuration using:

source build/envsetup.sh
lunch

Select a device type and press Enter.

Build the harness using:

make afw-test-harness -j32

This creates a directory (out/host/linux-x86/afw-th/android-cts) with all necessary binaries, configuration files, and tools to run the test harness. This directory is also zipped into a file (out/host/linux-x86/afw-th/android-afw-test-harness.zip) for distribution.

Running the AE Test Harness

Use the following steps to run the AE Test Harness:

In your build environment, launch the test runner using:
```
afw-test-tradefed
```
This starts the cts-tf console, loads test plans, test cases, and afw-test.props from out/host/linux-x86/afw-th/android-cts.
From the unzipped folder of android-afw-test-harness.zip, launch the test runner using:
```
cts-tf> ./android‐cts/tools/afw-test‐tradefed
```
This loads test plans, test cases, and afw-test.props from android-cts directory. Ensure ./android‐cts/repository/testcases/afw-test.props has the work account and Wi-Fi configuration.
Run a test plan. Each test plan is an XML file that contains a set of test packages from the AfwTestHarness/tests test package directory. Common plans include:
- afw-userdebug-build. Contains all test packages that require a userdebug build.
- afw-user-build. Runs on a user build but requires the test device to be set up properly, including completing the initial setup and enabling USB debugging.
To run the test plan afw-userdebug-build, use:
```
cts-tf> run cts --plan afw-userdebug-build
```
To see all test plans, use the command list plans. To view plan definitions, refer to out/host/linux-x86/afw-th/android-cts/repository/plans.
Run a test package. To run a single test package, use
```
cts-tf> run cts --package com.android.afwtest.NfcProvisioning
```
To view all packages, use the command list packages. For more options, use the command run cts --help.

Debugging the AE Test Harness

Run all commands in the afw-test-tradefed console (cts-tf), which you can launch by running afw-test-tradefed.

Display more information with the -l INFO or -l DEBUG flags. Example:
```
cts-tf> run cts ‐‐plan afw-userdebug-build -l DEBUG
```

Run the test harness on a specific device with the -s flag. Example:

cts-tf> run cts ‐‐plan afw-userdebug-build -l DEBUG -s device_sn

Run test harness on all connected devices with the --all-devices flag. Example:
```
cts-tf> run cts ‐‐plan afw-userdebug-build -l DEBUG --all-devices
```
View current running executions using list invocations or l i.
View summary of past test executions using list results or l r.
View other list commands using help list.
Monitor real-time logcat with filter using afwtest, then open another terminal and start logcat using: adb logcat | grep afwtest. After a test completes:
- View logs in out/host/linux-x86/afw-th/android-cts/repository/logs/start-time. The full device logcat and host log (afw-test-tradefed logs) are saved in separate zip files.
- Find relevant information by searching the device logcat for afwtest. Example: zless out/host/linux-x86/afw-th/android-cts/repository/logs/start-time/device_logcat_random-number.zip | grep afwtest
- To view the full afw-test-tradefed log, use: zless out/host/linux-x86/afw-th/android-cts/repository/logs/start-time/host_log_random-number.zip
A test package automates an enterprise provisioning flow by going through UI pages and recording a navigation log in the device logcat file for each page. Example: afwtest.AutomationDriver: Navigating:com.android.afwtest.uiautomator.pages.gms.AddAccountPage
UI pages for test package com.android.afwtest.NfcProvisioning include:
- com.android.afwtest.uiautomator.pages.managedprovisioning.NfcProvisioningPage
- com.android.afwtest.uiautomator.pages.PageSkipper
- com.android.afwtest.uiautomator.pages.LandingPage
If a test failed during the provisioning process, logcat contains an error similar to:
```
TestRunner: java.lang.RuntimeException: Failed to load page: com.android.afwtest.uiautomator.pages.packageinstaller.DeviceAccessPage
```
This is typically caused by errors in a previous UI page or the page that failed to load, so try to find other error messages in logcat before this error, then try to reproduce it manually following the provisioning flow.
If a test package fails:
- A screenshot is saved to out/host/linux-x86/afw-th/android-cts/repository/logs/start-time using the following syntax: screenshot-test_test_class_full_name_test_case_name-random_number.png. This information is also logged in the host log.
- A bug report is saved to out/host/linux-x86/afw-th/android-cts/repository/logs/start-time as: bug-test_class_full_name_test_case_name-random_number.zip.
After all test packages execute, a screenshot is taken and saved to out/host/linux-x86/afw-th/android-cts/repository/logs/start-time as: screenshot-random_number.png. This information is also logged in the host log.

FAQ

Can I run test plan afw-userdebug-build on a device flashed with user build?

No. Test packages in the afw-userdebug-build plan factory reset the testing device before running the actual test flow and require adb debugging to be auto-enabled. With a user build, adb debugging can be enabled only by manually changing the setting in Developer options.

Can I run test plan afw-user-build on a device flashed with userdebug build?

Yes, but we recommend that you run this test plan on a user build.

Sometimes my test fails because UI loading takes too much time. How can I fix this?

Configure the timeout_size setting in ./android-cts/repository/testcases/afw-test.props. Valid settings are: S, M, L, XL, XXL.

The test package com.android.afwtest.NfcProvisioning (or SuwDoProvisioning) fails on my device because the installed initial setup shows customized UI (such as Term & Conditions) after provisioning is complete. How can I skip this customized UI?

There should be minimal UI after the provisioning process. The test harness will automatically skip such UI if the UI has a button that has meaningful text or content description that contains any of the following words: Skip, Finish, Done, Accept, Agree, Next, Continue, or Proceed. Alternatively, you can define a button in afw-test.props to configure the test harness to skip your UI. Example:

oem_widgets=your_btn
your_btn.text=your_customized_text
your_btn.package=your_package
your_btn.action=click

To define multiple widgets, separate using commas.

The test package com.android.afwtest.NfcProvisioning (or SuwDoProvisioning) failed and the last UI screen is "Verify your account." Why does this happen and how can I recover the testing device?

This failure occurs because the previous test package failed to clear Factory Reset Protection at the end of the test. You must manually enter the account to unlock the device.

My device needs more time to factory reset. Can I extend the factory reset timeout?

Yes. Configure the factory_reset_timeout_min setting in afw-test.props. Valid settings are in minutes; you can set to any number of minutes that works with your device.