Collect performance data

Watchdog native service records I/O performance data since early-init during system boot up. The native service dumps the collected performance data in the bug report. Vendors can query the native service via dumpsys to either dump the collected data or start a custom collection session for debugging.

Data collection frequency

The frequency of the performance data collection varies by the build type.

  • On userdebug or eng builds, Watchdog collects performance data once every one (1) second during system boot up and once every one (1) minute post boot complete.
  • On user builds, Watchdog collects performance data once every 20 seconds during system boot up and once every two (2) minutes after a boot is complete.

Dump performance data

When a bugreport is captured, Watchdog native service dumps the collected performance data in the bug report. Vendors can query the native service via the below dumpsys command to dump the collected performance data. See the section on Understanding the performance data report for details about the report.

$ adb shell dumpsys android.automotive.watchdog.ICarWatchdog/default > /tmp/carwatchdog_dump.txt

Collect performance data for debugging

Vendors can use the Watchdog native service to custom collect performance data for debugging purposes. For example, to understand the I/O usage of an app or service, vendors can perform a custom performance data collection that profiles the I/O usage of either a specific list of packages or the entire system.

Start custom collection session

Shown below is the command to start a custom performance data collection session:

$ adb shell dumpsys android.automotive.watchdog.ICarWatchdog/default --start_perf [--interval <seconds>] [--max_duration <seconds>] [--filter_packages <package_name>,<package_name>,...]
  • --start_perf. Starts a custom performance data collection session.
  • --interval <seconds>. Specifies the profiling interval. By default, the interval is 30 seconds.
  • --max_duration <seconds>. Specifies the maximum duration of the custom performance data collection session. If the session isn't stopped manually, the session is stopped and the collected data is discarded after this duration. By default, the maximum duration is 30 minutes.
  • --filter_packages <package_name>,<package_name>,.... Specifies a comma-separated list of package names to profile. When provided, the performance data collection is limited to these packages. Otherwise, the performance data collection is done for all packages in the system.

For example, the below command starts a custom performance data collection with a profiling interval of 10 seconds, maximum collection duration of 1 hour, and limits the profiling to the com.google.android.car.kitchensink and com.google.android.apps.maps packages.

$ adb shell dumpsys android.automotive.watchdog.ICarWatchdog/default --start_perf --interval 10 --max_duration 3600 --filter_packages com.google.android.car.kitchensink,com.google.android.apps.maps

Stop and dump custom collection session

After recording the I/O usage of an app or service of interest, vendors should stop the custom performance data collection session to dump the collected data.

The following command stops the custom performance data collection and dumps the collected data to /tmp/carwatchdog_dump.txt:

$ adb shell dumpsys android.automotive.watchdog.ICarWatchdog/default --stop-perf > /tmp/carwatchdog_custom_dump.txt

Dumpsys help

$ adb shell dumpsys android.automotive.watchdog.ICarWatchdog/default --help

Understand the performance data report

Shown below is a sample performance data report:

$ adb shell dumpsys android.automotive.watchdog.ICarWatchdog/default

------------------------------
WatchdogPerfService report:
------------------------------
Boot-time collection information:
==========================
Event interval: 1 second

Periodic collection information:
=========================
Event interval: 60 seconds
--------------------------------
Boot-time I/O performance report:
============================
Collection duration: 1637609138 seconds
Number of collections: 12

Collection 0: <Fri Jan  7 19:53:53 2022 GMT>
===================================
CPU I/O wait time/percent: 18 / 0.63%
Number of I/O blocked processes/percent: 0 / 0.00%

Top N Reads:
-------------
Android User ID, Package Name, Foreground Bytes, Foreground Bytes %, Foreground Fsync, Foreground Fsync %, Background Bytes, Background Bytes %, Background Fsync, Background Fsync %
0, root, 56123392, 81.95%, 1, 100.00%, 0, 0.00%, 0, 0.00%
0, system, 12333056, 18.01%, 0, 0.00%, 0, 0.00%, 0, 0.00%

Top N Writes:
-------------
Android User ID, Package Name, Foreground Bytes, Foreground Bytes %, Foreground Fsync, Foreground Fsync %, Background Bytes, Background Bytes %, Background Fsync, Background Fsync %
0, root, 53248, 100.00%, 1, 100.00%, 0, 0.00%, 0, 0.00%

Top N I/O waiting UIDs:
-----------------------
Android User ID, Package Name, Number of owned tasks waiting for I/O, Percentage of owned tasks waiting for I/O
        Command, Number of I/O waiting tasks, Percentage of UID's tasks waiting for I/O
0, root, 2, 0.59%
        kworker/7:2+events, 1, 50.00%
        kworker/4:2+events, 1, 50.00%
0, system, 2, 7.14%
        ais_server, 2, 100.00%

Top N major page faults:
------------------------
Android User ID, Package Name, Number of major page faults, Percentage of total major page faults
        Command, Number of major page faults, Percentage of UID's major page faults
0, system, 48, 53.33%
        ais_server, 13, 27.08%
        audiod, 10, 20.83%
0, root, 41, 45.56%
        init, 39, 95.12%
        ueventd, 1, 2.44%
0, media, 1, 1.11%
        init.qti.media., 1, 100.00%
Number of major page faults since last collection: 90
Percentage of change in major page faults since last collection: 0.00%

Collection 1: <Fri Jan  7 19:53:54 2022 GMT>
===================================
 … Collection data …

Collection 2: <Fri Jan  7 19:53:55 2022 GMT>
===================================
 … Collection data …
 … Repeated collection data until boot-complete …

-----------------------------------------
Last N minutes I/O performance report:
================================
Collection duration: 10797 seconds
Number of collections: 181

Collection 0: <Tue Jan 11 19:43:05 2022 GMT>
====================================
CPU I/O wait time/percent: 18 / 0.04%
Number of I/O blocked processes/percent: 0 / 0.00%

Top N Writes:
-------------
Android User ID, Package Name, Foreground Bytes, Foreground Bytes %, Foreground Fsync, Foreground Fsync %, Background Bytes, Background Bytes %, Background Fsync, Background Fsync %
0, system, 33697792, 92.41%, 0, 0.00%, 0, 0.00%, 0, 0.00%
0, gps, 2420736, 6.64%, 0, 0.00%, 0, 0.00%, 0, 0.00%

Top N I/O waiting UIDs:
-----------------------
Android User ID, Package Name, Number of owned tasks waiting for I/O, Percentage of owned tasks waiting for I/O
        Command, Number of I/O waiting tasks, Percentage of UID's tasks waiting for I/O
0, root, 1, 0.19%
        crtc_commit:244, 1, 100.00%

When capturing a bugreport, running a Watchdog dumpsys command without any arguments or stopping a custom collection session, the Watchdog native service dumps a performance data report similar to the above report.

The report contains data collected during system boot up and the last N minutes before report generation.

  • Boot-time collection information. Provides information about the boot-time performance data collection session.
  • Periodic collection information. Provides information about the periodic performance collection session that starts post boot-complete.
  • Boot-time I/O performance report. Contains performance data collected during system boot-up. This data is persistent in-memory until system shutdown, so the data is dumped on all reports.
  • Last N minutes I/O performance report. Contains performance data collected (by the periodic collection session) during the last N minutes before the report generation.
  • Collection duration. Total duration of the performance data collection.
  • Number of collections. Total number of collections reported in a single report.

Each collection reported in the report may contain all or some of the below sections. The stats reported in each collection is the delta since the last collection except the stats that are instantaneous (for example, not aggregated since system boot-up).

  • CPU I/O wait time/percent. Absolute CPU time and percentage of total CPU time spent on context switching or waiting due to I/O operations since the last collection.
  • Number of I/O blocked processes/percent. Total number of processes blocked waiting for I/O and percentage of processes blocked waiting for I/O since the last collection.
  • Top N Reads. Top N packages that read most data from disk since the last collection.
    • Each line contains the user ID, package name, total number of bytes read in foreground mode vs background mode, percentage of bytes across all foreground mode vs background mode reads in the system, total number of fsync calls made in foreground mode vs background mode, and percentage of fsync calls across all foreground mode vs background mode fsync calls made in the system.
  • Top N Writes. Top N packages that wrote most data to disk since the last collection.
    • Each line contains fields similar to that of Top N Reads.
  • Top N I/O waiting UIDs. Top N packages with the most I/O waiting tasks.
    • Each top-level line contains the user ID, package names, number and percentage of tasks owned by the package waiting for I/O.
    • Inner-level lines contain the I/O waiting information for the top processes owned by the package.
  • Top N major page faults. Top N packages with the most major page faults since the last collection.
    • Each top-level line contains the user ID, package names, number and percentage of major page faults attributed to the package.
    • Inner-level lines contain the major page faults information for the top processes owned by the package.
  • Number of major page faults since last collection.
  • Percentage of change in major page faults since last collection.