Test code within feature launch flags

With the introduction of feature launch flags, there are new testing policies that you must adhere to:

Your tests must cover both enabled and disabled behaviors of the flag.
You must use the official mechanisms to set flag values during testing.
xTS tests shouldn't override flag values in tests.

The next section provides the official mechanisms you must use to adhere to these policies.

Test your flagged code

Test scenario	Mechanism used
Local testing when flag values change often	Android debug bridge as discussed in Change a flag's value at runtime
Local testing when flag values don't change often	Flag values file as discussed in Set feature launch flag values
End-to-end testing where flag values change	`FeatureFlagTargetPreparer` as discussed in Create end-to-end tests
Unit testing where flag values change	`SetFlagsRule` with `@EnableFlags` and `@DisableFlags` as discussed in Create unit tests (Java and Kotlin) or Create unit tests (C and C++)
End-to-end or unit testing where flag values can't change	`CheckFlagsRule` as discussed in Create end-to-end or unit tests where flag values don't change

Create end-to-end tests

AOSP provides a class called FeatureFlagTargetPreparer, which enables end-to-end testing on a device. This class accepts flag value overrides as input, sets those flags in the devices configuration before the test execution, and restores flags after execution.

You can apply the functionality of the FeatureFlagTargetPreparer class at the test module and test config levels.

Apply FeatureFlagTargetPreparer in a test module configuration

To apply FeatureFlagTargetPreparer in a test module configuration, include FeatureFlagTargetPreparer and flag value overrides in the AndroidTest.xml test module configuration file:

  <target_preparer class="com.android.tradefed.targetprep.FeatureFlagTargetPreparer">
        <option name="flag-value"
            value="permissions/com.android.permission.flags.device_aware_permission_grant=true"/>
        <option name="flag-value"
            value="virtual_devices/android.companion.virtual.flags.stream_permissions=true"/>
    </target_preparer>

Where:

target.preparer class is always set to com.android.tradefed.targetprep.FeatureFlagTargetPreparer.
option is the flag override with name always set to flag-value and value set to namespace/aconfigPackage.flagName=true|false.

Create parameterized test modules based on flag states

To create parameterized test modules based on flag states:

Include FeatureFlagTargetPreparer in the AndroidTest.xml test module configuration file:

<target_preparer class="com.android.tradefed.targetprep.FeatureFlagTargetPreparer" >

Specify flag value options in the test_module_config section of an Android.bp build file:

android_test {
    name: "MyTest"
    ...
}

test_module_config {
    name: "MyTestWithMyFlagEnabled",
    base: "MyTest",
    ...
    options: [
        {name: "flag-value", value: "telephony/com.android.internal.telephony.flags.oem_enabled_satellite_flag=true"},
    ],
}

test_module_config {
    name: "MyTestWithMyFlagDisabled",
    base: "MyTest",
    ...
    options: [
        {name: "flag-value", value: "telephony/com.android.internal.telephony.flags.carrier_enabled_satellite_flag=true"},
    ],
}

The options field contains the flag overrides with name always set to flag-value and value set to namespace/aconfigPackage.flagName=true|false.

Create unit tests (Java and Kotlin)

This section describes the approach to overriding aconfig flag values at the class and method level (per-test) in Java and Kotlin tests.

To write automated unit tests in a large codebase with a large number of flags, follow these steps:

Use the SetFlagsRule class with the @EnableFlags and @DisableFlags annotations to test all code branches.
Use the SetFlagsRule.ClassRule method to avoid common test bugs.
Use FlagsParameterization to test your classes across a broad set of flag configurations.

Test all code branches

For projects that use the static class to access flags, the SetFlagsRule helper class is provided to override flag values. The following code snippet shows how to include the SetFlagsRule and enable several flags at once:

  import android.platform.test.annotations.EnableFlags;
  import android.platform.test.flag.junit.SetFlagsRule;
  import com.example.android.aconfig.demo.flags.Flags;
  ...
    @Rule public final SetFlagsRule mSetFlagsRule = new SetFlagsRule();

    @Test
    @EnableFlags({Flags.FLAG_FLAG_FOO, Flags.FLAG_FLAG_BAR})
    public void test_flag_foo_and_flag_bar_turned_on() {
    ...
    }

Where:

@Rule is an annotation used to add the flag-JUnit dependency of the SetFlagsRule class.
SetFlagsRule is helper class provided to override flag values. For information on how SetFlagsRule determines default values, see Device default values.
@EnableFlags is an annotation that accepts an arbitrary number of flag names. When disabling flags, use @DisableFlags. You can apply these annotations to either a method or a class.

Set flag values for the entire test process, starting with the SetFlagsRule, which is prior to any @Before-annotated setup methods in the test. Flag values return to their previous state when the SetFlagsRule finishes, which is after any @After-annotated setup methods.

Ensure flags are set correctly

As mentioned previously, SetFlagsRule is used with the JUnit @Rule annotation, which means that SetFlagsRule can't ensure your flags are set correctly during the test class's constructor, or any @BeforeClass or @AfterClass-annotated methods.

To ensure that test fixtures are constructed with the correct class value, use the SetFlagsRule.ClassRule method so your fixtures aren't created until an @Before-annotated setup method:

  import android.platform.test.annotations.EnableFlags;
  import android.platform.test.flag.junit.SetFlagsRule;
  import com.example.android.aconfig.demo.flags.Flags;

  class ExampleTest {
    @ClassRule public static final SetFlagsRule.ClassRule mClassRule = new SetFlagsRule.ClassRule();
    @Rule public final SetFlagsRule mSetFlagsRule = mClassRule.createSetFlagsRule();

    private DemoClass underTest = new DemoClass();

    @Test
    @EnableFlags(Flags.FLAG_FLAG_FOO)
    public void test_flag_foo_turned_on() {
      ...
    }
  }

By adding the SetFlagsRule.ClassRule class rule, test_flag_foo_turned_on fails before running when FLAG_FLAG_FOO is read by the constructor of DemoClass.

If your entire class needs a flag enabled, move the @EnableFlags annotation to the class level (before the class declaration). Moving the annotation to the class level allows SetFlagsRule.ClassRule to ensure the flag is set correctly during the test class's constructor, or during any @BeforeClass or @AfterClass-annotated methods.

Run tests across multiple flag configurations

Because you can set flag values on a per-test basis, you can also use parameterization to run tests across multiple flag configurations:

...
import com.example.android.aconfig.demo.flags.Flags;
...

@RunWith(ParameterizedAndroidJunit4::class)
class FooBarTest {
    @Parameters(name = "{0}")
    public static List<FlagsParameterization> getParams() {
        return FlagsParameterization.allCombinationsOf(Flags.FLAG_FOO, Flags.FLAG_BAR);
    }

    @Rule
    public SetFlagsRule mSetFlagsRule;

    public FooBarTest(FlagsParameterization flags) {
        mSetFlagsRule = new SetFlagsRule(flags);
    }

    @Test public void fooLogic() {...}

    @DisableFlags(Flags.FLAG_BAR)
    @Test public void legacyBarLogic() {...}

    @EnableFlags(Flags.FLAG_BAR)
    @Test public void newBarLogic() {...}
}

Note that with SetFlagsRule, but without parameterization, this class runs three tests (fooLogic, legacyBarLogic, and newBarLogic). The fooLogic method runs with whatever the values of FLAG_FOO and FLAG_BAR are set to on the device.

When parameterization is added, the FlagsParameterization.allCombinationsOf method creates all possible combinations of the FLAG_FOO and FLAG_BAR flags:

FLAG_FOO is true and FLAG_BAR is true
FLAG_FOO is true and FLAG_BAR is false
FLAG_FOO is false and FLAG_BAR is true
FLAG_FOO is false and FLAG_BAR is false

Instead of directly changing flag values, @DisableFlags and @EnableFlags annotations modify flag values based on parameter conditions. For example, legacyBarLogic runs only when FLAG_BAR is disabled, which occurs in two of the four flag combinations. The legacyBarLogic is skipped for the other two combinations.

There are two methods for creating the parameterizations for your flags:

FlagsParameterization.allCombinationsOf(String...) executes 2^n runs of each test. For example, one flag runs 2x tests or four flags run 16x tests.
FlagsParameterization.progressionOf(String...) executes n+1 runs of each test. For example, one flag runs 2x tests and four flags run 5x flags.

Create unit tests (C and C++)

AOSP includes flag value macros for C and C++ tests written in the GoogleTest framework.

In your test source, include the macro definitions and aconfig-generated libraries:
```
#include <flag_macros.h>
#include "android_cts_flags.h"
```

In your test source, instead of using TEST and TESTF macros for your test cases, use TEST_WITH_FLAGS and TEST_F_WITH_FLAGS:

#define TEST_NS android::cts::flags::tests

...

TEST_F_WITH_FLAGS(
  TestFWithFlagsTest,
  requies_disabled_flag_enabled_skip,
  REQUIRES_FLAGS_DISABLED(ACONFIG_FLAG(TEST_NS, readwrite_enabled_flag))
) {
  TestFail();
}

...

TEST_F_WITH_FLAGS(
  TestFWithFlagsTest,
  multi_flags_for_same_state_skip,
  REQUIRES_FLAGS_ENABLED(
      ACONFIG_FLAG(TEST_NS, readwrite_enabled_flag),
      LEGACY_FLAG(aconfig_flags.cts, TEST_NS, readwrite_disabled_flag)
  )
) {
  TestFail();
}

...

TEST_WITH_FLAGS(
  TestWithFlagsTest,
  requies_disabled_flag_enabled_skip,
  REQUIRES_FLAGS_DISABLED(
      LEGACY_FLAG(aconfig_flags.cts, TEST_NS, readwrite_enabled_flag))
) {
  FAIL();
}

...

TEST_WITH_FLAGS(
  TestWithFlagsTest,
  requies_enabled_flag_enabled_executed,
  REQUIRES_FLAGS_ENABLED(ACONFIG_FLAG(TEST_NS, readwrite_enabled_flag))
) {
  TestWithFlagsTestHelper::executed_tests.insert(
      "requies_enabled_flag_enabled_executed");
}

Where:

TEST_WITH_FLAGS and TEST_F_WITH_FLAGS macros are used instead of TEST and TEST_F macros.
REQUIRES_FLAGS_ENABLED defines a set of feature release flags that must meet the enabled condition. You can write these flags in ACONFIG_FLAG or LEGACY_FLAG macros.
REQUIRES_FLAGS_DISABLED defines a set of feature flags that must meet the disabled condition. You can write these flags in ACONFIG_FLAG or LEGACY_FLAG macros.
ACONFIG_FLAG (TEST_NS, readwrite_enabled_flag) is a macro used for flags defined in aconfig files. This macro accepts a namespace (TEST_NS) and a flag name (readwrite_enabled_flag).
LEGACY_FLAG(aconfig_flags.cts, TEST_NS, readwrite_disabled_flag) is a macro used for flags set in device config by default.

In your Android.bp build file, add the aconfig-generated libraries and relevant macro libraries as a test dependency:

cc_test {
  name: "FlagMacrosTests",
  srcs: ["src/FlagMacrosTests.cpp"],
  static_libs: [
      "libgtest",
      "libflagtest",
      "my_aconfig_lib",
  ],
  shared_libs: [
      "libbase",
      "server_configurable_flags",
  ],
  test_suites: ["general-tests"],
  ...
}

Run the tests locally with this command:
```
atest FlagMacrosTests
```
If the flag my_namespace.android.myflag.tests.my_flag is disabled, the test result is:
```
[1/2] MyTest#test1: IGNORED (0ms)
[2/2] MyTestF#test2: PASSED (0ms)
```
If the flag my_namespace.android.myflag.tests.my_flag is enabled, the test result is:
```
[1/2] MyTest#test1: PASSED (0ms)
[2/2] MyTestF#test2: IGNORED (0ms)
```

Create end-to-end or unit tests where flag values don't change

For test cases where you can't override flags and can filter tests only if they're based on the current flag state, use the rule CheckFlagsRule with RequiresFlagsEnabled and RequiresFlagsDisabled annotations.

The following steps show you how to create and run an end-to-end or unit test where flag values can't be overridden:

In your test code, use CheckFlagsRule to apply test filtering. Also, use the Java annotations RequiresFlagsEnabled and RequiredFlagsDisabled to specify the flag requirements for your test.

The device-side test uses the DeviceFlagsValueProvider class:

@RunWith(JUnit4.class)
public final class FlagAnnotationTest {
  @Rule
  public final CheckFlagsRule mCheckFlagsRule =
          DeviceFlagsValueProvider.createCheckFlagsRule();

  @Test
  @RequiresFlagsEnabled(Flags.FLAG_FLAG_NAME_1)
  public void test1() {}

  @Test
  @RequiresFlagsDisabled(Flags.FLAG_FLAG_NAME_1)
  public void test2() {}
}

The host-side test uses the HostFlagsValueProvider class:

@RunWith(DeviceJUnit4ClassRunner.class)
public final class FlagAnnotationTest extends BaseHostJUnit4Test {
  @Rule
  public final CheckFlagsRule mCheckFlagsRule =
          HostFlagsValueProvider.createCheckFlagsRule(this::getDevice);

  @Test
  @RequiresFlagsEnabled(Flags.FLAG_FLAG_NAME_1)
  public void test1() {}

  @Test
  @RequiresFlagsDisabled(Flags.FLAG_FLAG_NAME_1)
  public void test2() {}
}

Add jflag-unit and aconfig-generated libraries to the static_libs section of the build file for your test:

android_test {
    name: "FlagAnnotationTests",
    srcs: ["*.java"],
    static_libs: [
        "androidx.test.rules",
        "my_aconfig_lib",
        "flag-junit",
        "platform-test-annotations",
    ],
    test_suites: ["general-tests"],
}

Use the following command to run the test locally:

atest FlagAnnotationTests

If the flag Flags.FLAG_FLAG_NAME_1 is disabled, the test result is:

[1/2] com.cts.flags.FlagAnnotationTest#test1: ASSUMPTION_FAILED (10ms)
[2/2] com.cts.flags.FlagAnnotationTest#test2: PASSED (2ms)

Otherwise the test result is:

[1/2] com.cts.flags.FlagAnnotationTest#test1: PASSED (2ms)
[2/2] com.cts.flags.FlagAnnotationTest#test2: ASSUMPTION_FAILED (10ms)

Device default values

The initialized SetFlagsRule uses flag values from the device. If the flag value on the device isn't overridden, such as with adb, then the default value is the same as the release configuration of the build. If the value on the device has been overridden, then SetFlagsRule uses the override value as the default.

If the same test is executed under different release configurations, the value of flags not explicitly set with SetFlagsRule can vary.

After each test, SetFlagsRule restores the FeatureFlags instance in Flags to its original FeatureFlagsImpl, so that it doesn't have side effects on other test methods and classes.