Monitor flash memory usage

Watchdog monitors flash memory usage by tracking the total amount of disk I/O writes made by all apps and services using the per-UID disk I/O stats exposed by the Kernel at the location `/proc/uid_io/stats`. When an app or service exceeds the disk I/O overuse threshold, Watchdog takes actions on the app or service. The disk I/O overuse thresholds and the action to take on overuse is predefined in the disk I/O overuse configuration.

Overuse thresholds

  • The disk I/O overuse thresholds are enforced on a daily basis, that is, all writes made by an app/service are aggregated since the beginning of the current UTC calendar day and checked against the thresholds defined in the overuse configurations.
  • When a vehicle is started multiple times on a given day, the Watchdog module stores the disk I/O usage stats on flash memory and aggregates them since the beginning of the current UTC calendar day.

Overuse actions

When an app repeatedly exceeds the defined disk I/O overuse thresholds, Watchdog takes actions defined in the overuse configuration.

  • All vendor apps and services are considered critical for the overall system stability, so they are not terminated on disk I/O overuse. However, the overuse configuration can define a list of safe-to-terminate vendor apps and services.
  • All third-party apps are safe-to-terminate.

When an app or service is safe-to-terminate, Watchdog disables the app or service with the app component state PackageManager.COMPONENT_ENABLED_STATE_DISABLED_UNTIL_USED.

Overuse configuration

The overuse configuration contains the disk I/O overuse thresholds and actions. Default overuse configurations are defined in the system and vendor images, and shipped with the build. Vendors can optionally include the vendor configuration in the vendor image. When the vendor configuration is not provided, the system configuration is used for the vendor apps and services as well.

Watchdog exposes system APIs through CarWatchdogManager, which lets vendors apps or services update the vendor configuration anytime.

Overuse configuration definition

Overuse configuration is split by the component type, for example, system, vendor, and third party. OEMs must update only the vendor component configuration.

Vendor configuration

Vendor configuration defines the disk I/O overuse thresholds and actions for all vendor apps and services, and all maps and media apps. The configuration contains the below configuration fields.

  • Vendor package prefixes. All packages installed in the vendor partition are considered as vendor packages. In addition to these packages, vendors can classify preinstalled packages as vendor packages by adding the package prefixes to the vendor package prefixes config. This config doesn't accept regular expressions.
  • Safe-to-terminate packages. Vendors can specify which vendor packages are safe to be terminated by adding the complete package names to the safe-to-terminate packages config.
  • Application category mappings. Vendors can map any package (including third-party packages) to one of the two supported app categories - Map and Media apps. This mapping is done to provide maps and media apps higher disk I/O overuse thresholds because these apps tend to download and write more data to disk than other app types.
  • Component level thresholds. Defines generic thresholds for all vendor packages (that is, packages not covered by Package specific thresholds or Application category specific thresholds get these thresholds). Vendors must define non-zero component-level thresholds when defining disk I/O overuse configuration.
  • Package specific thresholds. Vendors can define special thresholds for specific vendor packages. The mappings should contain the complete package names. The thresholds defined in this config takes precedence over thresholds defined in other configs for a given package.
  • Application category specific thresholds. Vendors can specify special thresholds for specific app categories. The app categories must be one of the supported categories - Maps and Media apps. The thresholds defined in this config are mapped to specific packages using Application category mappings.
  • System-wide thresholds. Vendors must not specify this config.

Vendor package prefixes, Safe-to-terminate packages, Component level thresholds, and Package specific thresholds configs are updatable only by the vendor configuration for vendor apps and services. Application category specific thresholds config can be updated only by the vendor configuration for all maps and media apps.

The overuse thresholds contain the amount of bytes allowed to be written during

  • An app/service foreground mode versus background mode
  • and system garage mode.

This classification allows user facing foreground apps/services to write more data than background apps/services. In Garage mode, apps and services tend to download updates, so each needs a higher threshold than apps and services running in other modes.

System and third-party configurations

OEMs should not update the system and third-party configurations.

  • System configuration defines I/O overuse thresholds and actions for system apps and services.
    • This configuration can also update the Application category mappings. Thus, this config field is shared between system and vendor configurations.
  • Third-party configuration defines thresholds for all third-party apps. All apps that are not preinstalled in the system are third-party apps.
    • All third-party apps receive the same thresholds (for example, no third-party app receives special thresholds) except maps and media apps, whose thresholds are defined by the vendor configuration.
    • The below disk I/O overuse thresholds are the default thresholds for the third-party apps. These thresholds are shipped with the system image.
      • 3 GiB write in the app foreground mode.
      • 2 GiB write in the app background mode.
      • 4 GiB write in the system garage mode.
    • These are base thresholds. These thresholds are updated as we understand disk I/O usage better.

Overuse configuration XML format

Default vendor configuration can be placed (this is optional) at the location /vendor/etc/automotive/watchdog/resource_overuse_configuration.xml in the build image. When this configuration isn't specified, the system-defined configuration is applied for vendor apps and services as well.

The XML file should contain only one tag for each config field. I/O overuse configuration must be defined in the XML file. All threshold values should be specified in the MiB unit.

A sample XML configuration is provided below:

<resourceOveruseConfiguration version="1.0">
      <componentType> VENDOR </componentType>

      <!-- List of safe to kill vendor packages. -->
      <safeToKillPackages>
            <package> com.vendor.package.A </package>
            <package> com.vendor.package.B </package>
      </safeToKillPackages>

      <!-- List of vendor package prefixes. -->
      <vendorPackagePrefixes>
            <packagePrefix> com.vendor.package </packagePrefix>
      </vendorPackagePrefixes>

      <!-- List of unique package names to app category mappings. -->
      <packagesToAppCategoryTypes>
            <packageAppCategory type="MEDIA"> com.vendor.package.A </packageAppCategory>
            <packageAppCategory type="MAPS"> com.google.package.B </packageAppCategory>
            <packageAppCategory type="MEDIA"> com.third.party.package.C </packageAppCategory>
      </packagesToAppCategoryTypes>

      <ioOveruseConfiguration>
        <!-- Thresholds in MiB for all vendor packages that don't have package specific thresholds. -->
            <componentLevelThresholds>
                  <state id="foreground_mode"> 1024 </state>
                  <state id="background_mode"> 512 </state>
                  <state id="garage_mode"> 3072 </state>
            </componentLevelThresholds>

            <packageSpecificThresholds>
                  <!-- IDs must be unique -->
                  <perStateThreshold id="com.vendor.package.C">
                    <state id="foreground_mode"> 400 </state>
                    <state id="background_mode"> 100 </state>
                    <state id="garage_mode"> 200 </state>
                  </perStateThreshold>

                  <perStateThreshold id="com.vendor.package.D">
                    <state id="foreground_mode"> 1024 </state>
                    <state id="background_mode"> 500 </state>
                    <state id="garage_mode"> 2048 </state>
                  </perStateThreshold>
            </packageSpecificThresholds>

            <!-- Application category specific thresholds. -->
            <appCategorySpecificThresholds>
                  <!-- One entry per supported application category -->
                  <perStateThreshold id="MEDIA">
                    <state id="foreground_mode"> 600 </state>
                    <state id="background_mode"> 700 </state>
                    <state id="garage_mode"> 1024 </state>
                  </perStateThreshold>

                  <perStateThreshold id="MAPS">
                    <state id="foreground_mode"> 800 </state>
                    <state id="background_mode"> 900 </state>
                    <state id="garage_mode"> 2048 </state>
                  </perStateThreshold>
            </appCategorySpecificThresholds>
      </ioOveruseConfiguration>
</resourceOveruseConfiguration>

Update overuse configuration through CarWatchdogManager system APIs

The above XML configuration can be provided only in the build image. If an OEM chooses to update the on-device configuration after a build is released, they can use the following APIs to make changes to the on-device configuration.

  • Grant the permission Car.PERMISSION_CONTROL_CAR_WATCHDOG_CONFIG to the caller.
  • Must use the existing configurations to update and set the new configurations. Use the API CarWatchdogManager.getResourceOveruseConfigurations to get the existing configurations. If existing configurations are not used, all configurations (including system and third-party configurations) are overwritten, which is not recommended.
  • Update the existing configurations with the delta changes and set the new configurations. Do not update the system and third-party component configurations.
  • Use the API CarWatchdogManager.setResourceOveruseConfigurations to set the new configurations.
  • To get and set the disk I/O overuse configurations use the flag CarWatchdogManager.FLAG_RESOURCE_OVERUSE_IO.

Here's a sample implementation that updates the resource overuse configurations:

void updateResourceOveruseConfigurations() {
    CarWatchdogManager manager =
        (CarWatchdogManager) car.getCarManager(Car.CAR_WATCHDOG_SERVICE);

    List<ResourceOveruseConfiguration> resourceOveruseConfigurations =
        manager.getResourceOveruseConfigurations(
            CarWatchdogManager.FLAG_RESOURCE_OVERUSE_IO);

    List<ResourceOveruseConfiguration> newResourceOveruseConfigurations =
            new List<>();
    ResourceOveruseConfiguration vendorConfiguration;
    for(ResourceOveruseConfiguration config : resourceOveruseConfigurations) {
        // Do not update the configurations of the system and third-party component types.
        if (config.getComponentType()
            != ResourceOveruseConfiguration.COMPONENT_TYPE_VENDOR) {
            newResourceOveruseConfigurations.add(config);
            continue;
        }
        vendorConfiguration = config;
    }

    if (vendorConfiguration == null) {
        ResourceOveruseConfiguration.Builder vendorConfigBuilder =
            new ResourceOveruseConfiguration.Builder();
        initializeConfig(vendorConfigBuilder);
        newResourceOveruseConfigurations.add(vendorConfigBuilder.build());
    } else {
        ResourceOveruseConfiguration newVendorConfig =
            updateConfig(vendorConfiguration);
        newResourceOveruseConfigurations.add(newVendorConfig);
    }
    int result = manager.setResourceOveruseConfigurations(
        newResourceOveruseConfigurations,

    if (result != CarWatchdogManager.RETURN_CODE_SUCCESS) {
        // Failed to set the resource overuse configurations.
    }
}

/** Sets the delta between the old configuration and the new configuration. */
ResourceOveruseConfiguration updateConfig(
    ResourceOveruseConfiguration oldConfiguration) {
    // Replace com.vendor.package.A with com.vendor.package.B in the safe-to-kill list.
    List<String> safeToKillPackages = oldConfiguration.getSafeToKillPackages();
    safeToKillPackages.remove("com.vendor.package.A");
    safeToKillPackages.add("com.vendor.package.B");

    ResourceOveruseConfiguration.Builder configBuilder =
        new ResourceOveruseConfiguration.Builder(
            oldConfiguration.getComponentType(),
            safeToKillPackages,
            oldConfiguration.getVendorPackagePrefixes(),
            oldConfiguration.getPackagesToAppCategoryTypes());

    configBuilder.addVendorPackagePrefixes("com.vendor.");
    configBuilder.addPackagesToAppCategoryTypes("com.vendor.package.B",
        ResourceOveruseConfiguration.APPLICATION_CATEGORY_TYPE_MAPS);

    IoOveruseConfiguration oldIoConfiguration = oldConfiguration.getIoOveruseConfiguration();
    IoOveruseConfiguration.Builder ioConfigBuilder =
        new IoOveruseConfiguration.Builder(
            oldIoConfiguration.getComponentLevelThresholds(),
            oldIoConfiguration.getPackageSpecificThresholds(),
            oldIoConfiguration.getAppCategorySpecificThresholds(),
            oldIoConfiguration.getSystemWideThresholds());

    // Define the amount of bytes based on the flash memory specification, expected lifetime,
    // and estimated average amount of bytes written by a package during different modes.
    ioConfigBuilder.addPackageSpecificThresholds("com.vendor.package.B",
        new PerStateBytes(/* foregroundModeBytes= */ 2 * 1024 * 1024 * 1024,
                          /* backgroundModeBytes= */ 500 * 1024 * 1024,
                          /* garageModeBytes= */ 3 * 1024 * 1024 * 1024));


    return configBuilder.setIoOveruseConfiguration(ioConfigBuilder.build()).build();
}

Apps monitoring their resource overuse

Vendor and third-party apps can listen for app specific resource overuse notifications from Watchdog or poll CarWatchdogManager for the app specific resource overuse statistics for up to the past 30 days.

Listen for resource overuse notifications

Apps can implement a resource overuse listener and register the listener with CarWatchdogManager to receive app specific notifications when they exceed 80% or 100% of their disk I/O overuse thresholds. Apps can use these notifications to:

  • Log the disk I/O overuse statistics for offline analysis. App developers can use this logging to debug the disk I/O overuse issue.
  • Reduce the disk I/O writes until the overuse counters reset.

Java client

  1. Implement listener by inheriting CarWatchdogManager.ResourceOveruseListener: 
    class ResourceOveruseListenerImpl implements
      CarWatchdogManager.ResourceOveruseListener {
            @Override
            public void onOveruse(
                  @NonNull ResourceOveruseStats resourceOveruseStats) {
                  // 1. Log/Upload resource overuse metrics.
                  // 2. Reduce writes until the counters reset.

                  IoOveruseStats ioOveruseStats = resourceOveruseStats.getIoOveruseStats();
                  // Stats period - [ioOveruseStats.getStartTime(), ioOveruseStats.getStartTime()
                  //   + ioOveruseStats.getDurationInSeconds()]
                  // Total I/O overuses - ioOveruseStats.getTotalOveruses()
                  // Total bytes written - ioOveruseStats.getTotalBytesWritten()
                  // Remaining write bytes for the current UTC calendar day -
                  //    ioOveruseStats.getRemainingWriteBytes()
            }
      }
}
  2. Register the listener instance by calling CarWatchdogManager.addResourceOveruseListener 
    private void addResourceOveruseListener() {
      CarWatchdogManager manager =
            (CarWatchdogManager) car.getCarManager(Car.CAR_WATCHDOG_SERVICE);
      // Choose a proper executor to handle resource overuse notifications.
      Executor executor = mContext.getMainExecutor();
      manager.addResourceOveruseListener(
            executor, CarWatchdogManager.FLAG_RESOURCE_OVERUSE_IO,
            mListenerImpl);
}
  3. Unregister the listener instance when the app has finished listening to: 
    private void removeResourceOveruseListener() {
    CarWatchdogManager manager =
            (CarWatchdogManager) car.getCarManager(Car.CAR_WATCHDOG_SERVICE);
    mCarWatchdogManager.removeResourceOveruseListener(
          mListenerImpl);
}

Native client

  1. Include carwatchdog_aidl_interface-ndk_platform in the shared_libs dependency of the build rule.

    Android.bp

    cc_binary {
    name: "sample_native_client",
    srcs: [
        "src/*.cpp"
    ],
    shared_libs: [
        "carwatchdog_aidl_interface-ndk_platform",
        "libbinder_ndk",
    ],
    vendor: true,
}
  2. Add SELinux policy to allow the vendor service domain to use binder (binder_user macro) and add the vendor service domain to the carwatchdog client domain (carwatchdog_client_domain macro). See the code below for sample_client.te and file_contexts.

    sample_client.te

    type sample_client, domain;
type sample_client_exec, exec_type, file_type, vendor_file_type;

carwatchdog_client_domain(sample_client)

init_daemon_domain(sample_client)
binder_use(sample_client)

    file_contexts

    /vendor/bin/sample_native_client  u:object_r:sample_client_exec:s0
  3. Implement the resource overuse listener by inheriting BnResourceOveruseListener. Override BnResourceOveruseListener::onOveruse to handle resource overuse notifications.

    ResourceOveruseListenerImpl.h

    class ResourceOveruseListenerImpl : public BnResourceOveruseListener {
public:
    ndk::ScopedAStatus onOveruse(
        ResourceOveruseStats resourceOveruseStats) override;

private:
    void initialize();
    void terminate();

    std::shared_ptr<ICarWatchdog> mWatchdogServer;
    std::shared_ptr<IResourceOveruseListener> mListener;
}

    ResourceOveruseListenerImpl.cpp

    ndk::ScopedAStatus ResourceOveruseListenerImpl::onOveruse(
      ResourceOveruseStats resourceOveruseStats) {

      // 1. Log/Upload resource overuse metrics.
      // 2. Reduce writes until the counters reset.

      if (stats.getTag() != ResourceOveruseStats::ioOveruseStats) {
            // Received resourceOveruseStats doesn't contain I/O overuse stats.
      }

      const IoOveruseStats& ioOveruseStats = stats.get();
      // Stats period - [ioOveruseStats.startTime,
      //   ioOveruseStats.startTime + ioOveruseStats.durationInSeconds]
      // Total I/O overuses - ioOveruseStats.totalOveruses
      // Total bytes written - ioOveruseStats.writtenBytes
      // Remaining write bytes for the current UTC calendar day -
      //    ioOveruseStats.remainingWriteBytes

      return ndk::ScopedAStatus::ok();
}
  4. Start a binder thread pool and register the resource overuse listener with the watchdog server. Watchdog server is registered under the service name android.automotive.watchdog.ICarWatchdog/default.

    main.cpp

    int main(int argc, char** argv) {
    ABinderProcess_setThreadPoolMaxThreadCount(1);
    ABinderProcess_startThreadPool();
    std::shared_ptr<ResourceOveruseListenerImpl> listener =
        ndk::SharedRefBase::make<ResourceOveruseListenerImpl>();

    // The listener is added in initialize().
    listener->initialize();

    ... Run service ...

    // The listener is removed in terminate().
    listener->terminate();
}

    ResourceOveruseListenerImpl.cpp

    void ResourceOveruseListener::initialize() {
    ndk::SpAIBinder binder(AServiceManager_getService(
            "android.automotive.watchdog.ICarWatchdog/default"));
    std::shared_ptr<ICarWatchdog> server = ICarWatchdog::fromBinder(binder);
    mWatchdogServer = server;

    std::shared_ptr<IResourceOveruseListener> listener =
        IResourceOveruseListener::fromBinder(this->asBinder());
    mWatchdogServer->addResourceOveruseListener(
      std::vector<int>{ResourceType.IO}, listener);
    mListener = listener;
}

void ResourceOveruseListener::terminate() {
    mWatchdogServer->removeResourceOveruseListener(mListener);
}

Poll resource overuse statistics

Apps can poll CarWatchdogManager for the app-specific I/O overuse statistics ATS for the most recent 30 days.

Java client

Use CarWatchdogManager.getResourceOveruseStats to get the resource overuse stats. Pass the CarWatchdogManager.FLAG_RESOURCE_OVERUSE_IO flag to get the disk I/O overuse stats.

private void getResourceOveruseStats() {
      CarWatchdogManager manager =
            (CarWatchdogManager) car.getCarManager(Car.CAR_WATCHDOG_SERVICE);

      // Returns resource overuse stats with I/O overuse stats for the past
      // 7 days. Stats are available for up to the past 30 days.
      ResourceOveruseStats resourceOveruseStats =
            mCarWatchdogManager.getResourceOveruseStats(
                  CarWatchdogManager.FLAG_RESOURCE_OVERUSE_IO,
                  CarWatchdogManager.STATS_PERIOD_PAST_7_DAYS);

      IoOveruseStats ioOveruseStats = resourceOveruseStats.getIoOveruseStats();
      // Stats period - [ioOveruseStats.getStartTime(), ioOveruseStats.getStartTime()
      //   + ioOveruseStats.getDurationInSeconds()]
      // Total I/O overuses - ioOveruseStats.getTotalOveruses()
      // Total bytes written - ioOveruseStats.getTotalBytesWritten()
      // Remaining write bytes for the UTC calendar day -
      //    ioOveruseStats.getRemainingWriteBytes()
}

Native client

Use CarWatchdogServer.getResourceOveruseStats to get the resource overuse stats. Pass the ResourceType.IO enum to fetch disk I/O overuse stats.

void getResourceOveruseStats() {
      ndk::SpAIBinder binder(AServiceManager_getService(
            "android.automotive.watchdog.ICarWatchdog/default"));
      std::shared_ptr<ICarWatchdog> server = ICarWatchdog::fromBinder(binder);
      // Returns the stats only for the current UTC calendar day.
      const std::vector<ResourceOveruseStats> resourceOveruseStats;
      ndk::ScopedAStatus status = server.getResourceOveruseStats(
            std::vector<int>{ResourceType.IO}, &resourceOveruseStats);
      if (!status.isOk()) {
            // Failed to get the resource overuse stats.
            return;
      }

      for (const auto& stats : resourceOveruseStats) {
            if (stats.getTag() != ResourceOveruseStats::ioOveruseStats) {
                  continue;
            }
            const IoOveruseStats& ioOveruseStats = stats.get();
            // Stats period - [ioOveruseStats.startTime,
            //   ioOveruseStats.startTime + ioOveruseStats.durationInSeconds]
            // Total I/O overuses - ioOveruseStats.totalOveruses
            // Total bytes written - ioOveruseStats.writtenBytes
            // Remaining write bytes for the current UTC calendar day -
            //   ioOveruseStats.remainingWriteBytes
      }
}

Resource overuse UX

Prioritize app performance setting

App settings page has the Prioritize app performance settings (see the image below), that allows the user to prioritize an app's performance over the system and long-term hardware performance. This setting is available only for apps that are safe to be terminated on resource overuse. Otherwise, this setting is grayed out. When this setting is toggled off (default setting) for an app, the app can be terminated on resource overuse. Otherwise, the app is not terminated on resource overuse.

When the user toggles on this setting, the following confirmation dialog describes the implications of toggling on the setting.

After 90 days, this setting is automatically reset to the default. The day limit can be modified with an RRO overlay app using watchdogUserPackageSettingsResetDays, up to a maximum of 180 days. To learn more, see Change the value of an app's resources at runtime. The following example overlay tag can be included in AndroidManifest.xml:

<overlay android:priority="<insert-value>"
       android:targetPackage="com.android.car.updatable"
       android:targetName="CarServiceCustomization"
       android:resourcesMap="@xml/overlays" />

In res/values/config.xml:

<resources>
  <integer name="watchdogUserPackageSettingsResetDays">value</integer>
</resources>

In res/xml/overlays.xml:

<overlay>
  <item target="integer/watchdogUserPackageSettingsResetDays" value="@integer/watchdogUserPackageSettingsResetDays" />
</overlay>

User notification

When an app or service repeatedly overuses disk I/O (for example, writes data to disk beyond the defined thresholds) within a certain period and is safe to be terminated on resource overuse, the user is notified after the vehicle enters the allow-driver-distraction state.

The first user notification (during a drive) is posted as a heads-up notification and the other notifications are posted on the notification center.

For example, when an app repeatedly overuses disk I/O, the user receives the following notification:

  • When the user clicks on the Prioritize app button, the app's settings page is launched, where the user can toggle on or off the Prioritize app performance setting.
  • When the user clicks on the Disable app button, the app is disabled until the user launches the app or enables it on the app's settings page.
  • For uninstallable apps, the Disable app button is replaced with the Uninstall app button. When the user clicks on the Uninstall app button, the app's Settings page is launched, from which the user can uninstall the app.

Recommendation for launcher implementation

When apps are disabled due to resource overuse, the apps disappear from the default launcher app because CarService updates the apps' enabled state as PackageManager.COMPONENT_ENABLED_STATE_DISABLED_UNTIL_USED. OEMs must update the builtin launcher implementation to display these apps as unusual, so users can use them if needed. See the following recommendations based on the build release.

Android SC V2 release