Collect kernel telemetry with eBPF

This page provides an overview of the Extended Berkeley Packet Filter (eBPF) kernel telemetry framework in Android, describing the framework architecture, built-in metrics, and how to add new kernel metrics.

Android includes a unified eBPF kernel telemetry framework for collecting low-overhead kernel metrics and exposing those metrics to userspace services such as statsd. By attaching portable eBPF programs to kernel tracepoints and kprobes, the framework aggregates high-frequency kernel events in place and exports compact histograms or counters without modifying the Generic Kernel Image (GKI) or breaking the Kernel Module Interface (KMI).

Traditional approaches to kernel telemetry rely on custom procfs or sysfs nodes, out-of-tree kernel patches, or vendor hooks that require custom plumbing across the kernel, SELinux, Java Native Interface (JNI), and userspace services. The eBPF kernel telemetry framework standardizes this pipeline across Android Common Kernels (ACKs) and provides the following benefits:

  • Avoids kernel ABI and KMI churn: Attaches eBPF programs to existing kernel tracepoints or kprobes without modifying kernel source code, changing the kernel application binary interface (ABI), or adding Android vendor hooks such as TRACE_ANDROID_VH_*. Device manufacturers and system-on-chip (SoC) partners can use eBPF telemetry in place of adding new telemetry vendor hooks or to migrate off existing vendor hooks.
  • Maintains portability across kernel versions: Uses BPF Compile Once - Run Everywhere (CO-RE) and BPF Type Format (BTF) so that a single compiled eBPF program adapts automatically to internal kernel struct layout changes across GKI versions, such as android14-6.1 and higher.
  • Minimizes system health overhead: Filters and aggregates high-frequency kernel events directly at the source into compact 32-bucket base-2 logarithmic (log2) histograms, typically under 1 KB per BPF map. Userspace retrieves aggregated map entries once per day using a single BPF_MAP_LOOKUP_BATCH batch syscall.
  • Simplifies SELinux and JNI integration: Pins telemetry maps under /sys/fs/bpf/kernelstats/ to apply directory-level fs_bpf_kernelstats SELinux type inheritance, and resolves struct member offsets at run time in JNI using BTF metadata to eliminate custom C++ parsing code when adding new metrics.

For more information about how Android loads and pins eBPF programs and maps, see Extend the kernel with eBPF.

Architecture overview

The eBPF kernel telemetry pipeline separates generic infrastructure components from metric-specific definitions so that adding a new kernel metric requires minimal code changes. The following table describes the components of the pipeline:

ComponentRole in pipelineScope
BPF loader, bpfloaderLoads compiled eBPF programs from /system/etc/bpf/kernelstats/ at boot if compatible with the running kernel and pins maps to /sys/fs/bpf/kernelstats/. Skips loading cleanly when disabled or unsupported.Generic
SELinux policy, fs_bpf_kernelstatsApplies directory-level SELinux labels to /sys/fs/bpf/kernelstats/ so system_server can read telemetry maps without per-metric SELinux policy updates.Generic
eBPF program in system/bpfprogs/telemetry/Attaches to kernel tracepoints or kprobes, filters events, and aggregates metrics into BPF maps.Custom per metric
JNI BTF readerParses BTF metadata from the compiled BPF Executable and Linkable Format (ELF) binary at run time, caches struct member offsets, and performs batch map reads into primitive Java arrays.Generic
Userspace puller in statsdPulls cumulative histogram maps on a daily schedule, computes deltas against cached history, and builds strongly typed StatsEvent pull atoms.Generic helper with a single case entry per atom
Telemetry schemaDefines the strongly typed statsd pull atom protobuf and server-side histogram or value metric aggregations.Custom per metric

Userspace consumption models

When designing a kernel telemetry metric, choose between two userspace consumption models depending on whether the metric represents cumulative state or an ordered event sequence. The following table compares the two models:

PropertyAccumulatorsStreams
DefinitionStateful, monotonically increasing cumulative counters or histogramsOrdered sequence of timestamped kernel events
Access patternUserspace polls on demand, such as once every 24 hours through statsdUserspace consumes continuously from a ring buffer to avoid dropped events
Typical BPF map typesBPF_MAP_TYPE_HASH, BPF_MAP_TYPE_LRU_HASH, BPF_MAP_TYPE_PERCPU_HASH, BPF_MAP_TYPE_LRU_PERCPU_HASHBPF_MAP_TYPE_RINGBUF
Concurrency modelKernel map lookups are atomic; multiple readers can poll without userspace lockingDestructive read semantics require a single reader thread dispatching to consumers
Example use casesMemory management latency histograms, Direct Memory Access Buffer (DMA-BUF) allocation sizes, wakeup source iteratorsFiltered scheduler trace events, high-frequency subsystem power traces

Built-in kernel telemetry metrics

Android includes built-in eBPF kernel telemetry programs that work across GKI 6.1 and higher kernels, including android14-6.1, android15-6.6, android16-6.12, and android17-6.18. Each latency and size metric uses a 32-bucket exponential (log2) histogram aggregated over a 24-hour window. The following table lists the built-in kernel telemetry metrics:

Metric atomSubsystemDescriptionDimension key
KswapdLatencyMemory managementKernel swap daemon, kswapd, wake-to-sleep execution latency in microsecondsNon-Uniform Memory Access (NUMA) node_id
DirectReclaimLatencyMemory managementSynchronous direct memory reclaim latency in microsecondsProcess out-of-memory (OOM) oom_score_adj state
CompactionDurationMemory managementSystem-wide memory compaction execution duration in microseconds, replacing legacy compaction vendor hooksSystem-wide
CmaAllocationLatencyMemory managementContiguous Memory Allocator (CMA) page allocation latency in microsecondsCMA heap name
DmabufRequestSizesMemory managementDMA-BUF allocation request sizes in bytesExporter heap name
SuspendResumeLatencyPower managementLatency across kernel suspend and resume stages in microsecondsSuspend action stage

Add a kernel telemetry metric

To add a new accumulator metric to the eBPF kernel telemetry pipeline, complete the following steps in order:

  1. Write the eBPF program under system/bpfprogs/telemetry/ and build the binary into /system/etc/bpf/kernelstats/.
  2. Verify BTF struct layout compatibility for the generic JNI BTF reader.
  3. Register the pull atom in statsd.
  4. Define the pull atom protobuf schema.

The following sections use DirectReclaimLatency (directReclaim.c) as an example.

Write the eBPF program

In system/bpfprogs/telemetry/, include <bpf_telemetry.h> and define the map key and output accumulator map. Use integer identifiers or enums in BPF map keys rather than raw strings to minimize map memory footprint and preserve privacy:

#include <android_bpf_defs.h>
#include <bpf/bpf_core_read.h>
#include <bpf/bpf_helpers.h>
#include <bpf_telemetry.h>
#include <stdint.h>

struct direct_reclaim_key {
    uint32_t process_state;
};

#define DIRECT_RECLAIM_START_MAP_ENTRIES 128
#define DIRECT_RECLAIM_OUTPUT_MAP_ENTRIES 8

// Temporary map to store start time. Key: pid (thread_id), Value: start_time_ns
DEFINE_BPF_MAP_GRO(direct_reclaim_start_map, HASH, uint32_t, uint64_t,
                   DIRECT_RECLAIM_START_MAP_ENTRIES, AID_SYSTEM);

// Output map. Key: direct_reclaim_key, Value: accumulator_value (32 log2 buckets)
DEFINE_BPF_MAP_GRO(direct_reclaim_latency_map, HASH, struct direct_reclaim_key,
                   struct accumulator_value, DIRECT_RECLAIM_OUTPUT_MAP_ENTRIES,
                   AID_SYSTEM);

In the tracepoint handlers, record the start timestamp on entry, compute the log2 latency bucket on exit using GET_LATENCY_BUCKET, and increment the accumulator map entry with INCREMENT_ACCUMULATOR:

DEFINE_BPF_PROG("tp_btf/mm_vmscan_direct_reclaim_begin", AID_ROOT, AID_SYSTEM,
                tracepoint_vmscan_mm_vmscan_direct_reclaim_begin)(
        uint64_t* args) {
    (void)args;
    uint32_t pid = (uint32_t)bpf_get_current_pid_tgid();
    SAVE_FIRST_START_TIME(direct_reclaim_start_map, &pid);
    return 0;
}

DEFINE_BPF_PROG("tp_btf/mm_vmscan_direct_reclaim_end", AID_ROOT, AID_SYSTEM,
                tracepoint_vmscan_mm_vmscan_direct_reclaim_end)(
        uint64_t* args) {
    (void)args;
    uint32_t pid = (uint32_t)bpf_get_current_pid_tgid();
    uint32_t bucket = GET_LATENCY_BUCKET(direct_reclaim_start_map, &pid);
    if (bucket == BPF_TELEMETRY_INVALID_BUCKET) return 0;

    struct task_struct* task = (struct task_struct*)bpf_get_current_task();
    int oom_score_adj = BPF_CORE_READ(task, signal, oom_score_adj);
    enum process_state state = get_process_state(oom_score_adj);

    struct direct_reclaim_key key = {.process_state = (uint32_t)state};
    INCREMENT_ACCUMULATOR(direct_reclaim_latency_map, &key, bucket);
    return 0;
}

Verify BTF struct layout compatibility

You don't need to write custom JNI C++ code when adding a new histogram map as long as your map key and value structs use standard integer, enum, and fixed array types. Instead of casting BPF map entries to hardcoded C structs at compile time, the generic JNI reader resolves struct layouts dynamically at run time in three phases:

  • ELF-to-map association: The JNI layer parses BTF metadata directly from the compiled BPF ELF file on disk, such as /system/etc/bpf/kernelstats/directReclaim.bpf, using the libbpf btf__parse function.
  • BTF type resolution: On first access, the reader builds and caches a structural representation of the map's key and value C structs, calculating cumulative byte offsets for struct members, integer and enum byte widths, and array element sizes from the base pointer.
  • Batch lookup and serialization: When statsd pulls the map, the JNI layer issues a single BPF_MAP_LOOKUP_BATCH command to the sys_bpf syscall to read all hash entries in bulk and copies the resolved key IDs and bucket arrays directly into primitive Java long[] arrays using the SetLongArrayRegion JNI function.

If a BPF C struct changes in a future release, the recompiled BTF section updates automatically and the JNI reader adapts at run time without requiring JNI recompilation.

Register the pull atom in statsd

In StatsPullAtomService.java, the generic pullKernelTelemetry helper reads the pinned BPF map, computes bucket deltas against the previous pull's cached counts, and emits StatsEvent entries. Adding a new kernel telemetry metric requires only three small additions:

// 1. Dispatch the atom tag to pullKernelTelemetry() in onPullAtomLocked():
case FrameworkStatsLog.DIRECT_RECLAIM_LATENCY:
    return pullKernelTelemetry(atomTag, data);

// 2. Add the atom tag to the registered atoms list in registerKernelTelemetry():
int[] atoms = {
    FrameworkStatsLog.DIRECT_RECLAIM_LATENCY,
    // ...
};

// 3. Map the atom tag to its pinned BPF map path:
case FrameworkStatsLog.DIRECT_RECLAIM_LATENCY ->
    "/sys/fs/bpf/kernelstats/map_directReclaim_direct_reclaim_latency_map";

Define the pull atom protobuf

Define the strongly typed pull atom message in frameworks/proto_logging/ to match the dimension key and 32-bucket log2 histogram array produced by the BPF map:

message DirectReclaimLatency {
    enum ProcessState {
        PROCESS_STATE_UNSPECIFIED = 0;
        STATE_NATIVE = 1;
        STATE_SYSTEM = 2;
        STATE_PERSISTENT = 3;
        STATE_FOREGROUND = 4;
        STATE_VISIBLE = 5;
        STATE_PERCEPTIBLE = 6;
        STATE_SERVICE_BACKGROUND = 7;
        STATE_CACHED = 8;
    }
    optional ProcessState process_state = 1;
    repeated int32 duration_log2_buckets_micros = 2;
}