Effective in 2026, to align with our trunk stable development model and ensure platform stability for the ecosystem, we will publish source code to AOSP in Q2 and Q4. For building and contributing to AOSP, we recommend utilizing android-latest-release instead of aosp-main. The android-latest-release manifest branch will always reference the most recent release pushed to AOSP. For more information, see Changes to AOSP.

Debug SDV

This guide provides an overview of the essential tools and techniques for debugging services and applications that run on the SDV platform.

Connect to the device

The primary tool for connecting to and interacting with the device is the Android Debug Bridge (adb). Verify that adb is in your system's path.

Connect using the network

If the device is on the same network and you know its IP address, you can connect by Ethernet. Our system exclusively uses network connections for adb:

# Connect to the device using its IP address and default port
adb connect <device_ip_address>:5555

Connect to multiple devices

To manage multiple devices from a single workstation, you can connect to them through different IP addresses. If you run multiple VMs on a single host, you need to assign different ports to each device. This requires configuring the adb daemon on each SDV device to listen on a unique port, for example, 5555 for the first device and 5556 for the second. You can also configure the host to forward those ports to different guests:

# Connect to the first device
adb connect <device_1_ip_address>:5555

# Connect to a second device
adb connect <device_2_ip_address>:5555

# Connect to a second VM on first device on different port
adb connect <device_1_ip_address>:5556

# List all connected devices to verify
adb devices

When you run subsequent adb commands, you can target a specific device by using the -s flag with its full address (ip:port):

# Run a shell command on the second VM on first device in the above example
adb -s <device_1_ip_address>:5556 shell

View logs

When a service fails or behaves unexpectedly, check the logs first.

Use logcat (Android logs)

logcat displays logs from system services, agents, and service bundles:

# View all logs in real-time
adb logcat

# Filter logs by a specific tag (e.g., the name of your service)
adb logcat -s MyServiceTag

# Filter logs by priority (V: Verbose, D: Debug, I: Info, W: Warn, E: Error, F: Fatal)
# This shows only Error and Fatal messages for a specific tag
adb logcat MyServiceTag:E *:S

# Clear old logs before viewing new ones
adb logcat -c && adb logcat

# Show all logs and return
adb logcat -d

# Color logs by level
adb logcat -v color

# Show timestamps as time since boot
adb logcat -v monotonic

Use dmesg (kernel logs)

dmesg displays logs from the Linux kernel. This is crucial for debugging the following:

Hardware and driver issues
Kernel panics
SELinux denials (avc: denied) - also displayed in logcat

# View all kernel messages
adb shell dmesg

# Follow kernel messages in real-time
adb shell dmesg -w

Access logs on Cuttlefish

When you run on a Cuttlefish virtual device, you can directly access the log files from the host machine's file system. This is especially useful if adb is not available. When you start Cuttlefish, it outputs all debugging paths, and you can also look them up using cvd fleet for all running instances.

Logcat (Android logs): cuttlefish/instances/cvd-1/logs/logcat
Kernel logs: cuttlefish/instances/cvd-1/logs/kernel.log
Cuttlefish launcher logs: cuttlefish/instances/cvd-1/logs/launcher.log (useful for debugging the Cuttlefish environment itself)

You can use standard command-line tools such as cat, less, or tail -f to view these files in real time.

Interact with the shell and file system

Gain shell access to run commands directly on the device, check process status, and navigate the file system:

# Open an interactive shell on the device
adb shell

# From within the shell, you can check running processes
ps -A

# Or check the status of a specific service managed by init
getprop init.svc.<service_name>

Perform file system operations

Use adb push and adb pull to move files between your host machine and the device. Use this to deploy test scripts, update configuration, or retrieve crash dumps:

# Push a file from your computer to the device
adb push my_local_file.txt /data/local/tmp/

# Pull a file from the device to your computer
adb pull /data/tombstones/tombstone_00 .

Debug applications (C++/Rust)

You can debug applications either online or offline.

Use lldb for live debugging

lldb is the default debugger in Android Studio and is often preferred for modern C++ and Rust development.

Follow these setup steps:

Restart adb to allow root permissions (required to forward). Use this to set up port forwarding. On the host, run:
```
adb root
```
If a binary has debugging symbols, push an unstripped version. You can find the unstripped version in out/target/product/[SDV_FLAVOR]/symbols/[PARTITION]/bin/[EXECUTABLE]. For example, to upload rpcagent from system_ext and sdv_core_cf, run:
```
adb push out/target/product/sdv_core_cf/symbols/system_ext/bin/rpcagent /data/local/tmp
```
Now the unstripped version is in /data/local/tmp.
Connect with the LLDB client using the lldbclient.py convenience script:
```
# Run and connect a binary on device, run on host:
lldbclient.py -r /data/local/tmp/rpcagent

# Connect to an existing process with PID 2759
lldbclient.py -p 2759
```
This script connects lldb remotely. For all available flags, run lldbclient.py. Refer to your IDE documentation to connect it to lldb.

Analyze crash dumps (tombstones)

When a service crashes, a tombstone file is generated in /data/tombstones/. The file contains a stack trace, memory map, and other crucial information.

Pull the tombstone file, then use a symbol-aware tool (like lldb or a dedicated script) to analyze it with the unstripped version of the binary.

Analyze performance

SDV supports multiple tools for performance measurements.

Check CPU and memory use

top: Provides a real-time view of running processes sorted by CPU usage.
```
adb shell top -m 10  # Show top 10 processes
```
procrank: Provides a detailed breakdown of memory usage for a specific process or the entire system.
```
adb shell procrank
```

Use dumpsys for service state

dumpsys is a powerful tool to query the internal state of any Android system service.

# List all available services
adb shell dumpsys -l

# Dump the state of a specific service or agent
adb shell dumpsys <service_name>

Use torq for performance analysis

torq is a command-line tool that simplifies profiling and tracing tasks on Android. It allows running Simpleperf, Perfetto, and other typical tasks during system performance analysis.

Use Perfetto for tracing

Perfetto is the standard tool for system-wide tracing on Android. Perfetto lets you record system events, app-level trace points, and performance counters, and visualize them on a timeline.

Key steps:

Add trace points in your C++ or Rust code:
- C++: Use the Perfetto SDK with TRACE_EVENT macros.
- Rust: Use the tracing crate, which emits ATrace events compatible with Perfetto.
Create a textproto configuration file to specify datasources (for example, ftrace, track_event), categories, and tags.
Use the record_android_trace script (external/perfetto/tools/record_android_trace) with your configuration file to capture the trace from the device.
Open the generated trace file in the Perfetto Web UI to analyze system behavior, identify bottlenecks, and understand service interactions.

For detailed instructions on instrumentation, configuration, and collection, see Use tracing to gain insights into system performance.

Use Simpleperf for profiling

Simpleperf is a versatile command-line tool for performance profiling on Android, similar to Linux perf.

Common use cases:

CPU profiling: Identify functions consuming the most CPU time.
Off-CPU analysis: Analyze why threads are sleeping or blocked.
Hardware counters: Access hardware performance counters.

Example commands:

# Go to simpleperf scripts
cd system/extras/simpleperf/scripts

# Profile system for 20 seconds and record call graph
./app_profiler.py --system_wide -r "--call-graph fp --duration 20"

# Record a profile for a specific process (PID 1)
./app_profiler.py --pid 1 -r "--call-graph fp --duration 5"

# Pull the profile data
adb pull /data/local/tmp/perf.data .

# Generate a report with symbol resolution
./report_html.py --add_source_code --source_dirs $ANDROID_BUILD_TOP --add_disassembly -o report.html

Simpleperf offers many more events and options. Refer to the Simpleperf guide for comprehensive documentation.

SELinux denials

Symptom: Your service fails to start, access a file, or use a capability, and you see an avc: denied message in dmesg or logcat.

[ 123.456] avc: denied { read } for pid=789 comm="my_service" name="some_file" dev="sda1" ino=123 scontext=u:r:my_service_t:s0 tcontext=u:object_r:some_file_t:s0 tclass=file permissive=0

Quick debug (for development only)

You can temporarily disable the SELinux enforcement to see if the issue is resolved. This confirms that the problem is due to a missing SELinux policy rule.

# Disabling SELinux requires root permission
adb root

# Set SELinux to permissive mode
adb shell setenforce 0

# Check the current mode (should return "Permissive")
adb shell getenforce

Fix: The audit2allow tool from the Android source tree on the denial message. This generates the correct .te policy rule to add to your device's SELinux configuration.