This section summarizes useful tools and related commands for debugging, tracing, and profiling native Android platform code when developing platform-level features.
Note: The pages in this section and elsewhere
within this site recommend the use of adb in conjunction with the
setprop argument to debug certain aspects of Android.
In Android 7.x and lower, property names had a length limit of 32
characters. This meant that to create a wrap property with the name of the app,
it was necessary to truncate the name to fit. In Android 8.0 and higher, this
limit is much greater and shouldn't require truncation.
This page covers the basics surrounding crash dumps found in logcat output.
Other pages have far more detail about
diagnosing native crashes,
exploring system services with
dumpsys, viewing
native memory,
network,
and RAM
usage, using AddressSanitizer to detect memory
bugs in native code, evaluating
performance issues (includes
systrace), and using
debuggers.
Crash dumps and tombstones
When a dynamically linked executable starts, several signal handlers are
registered that, in the event of a crash, cause a basic crash dump to be written to logcat
and a more detailed tombstone file to be written to /data/tombstones/.
The tombstone is a file with extra data about the crashed process. In particular, it contains
stack traces for all the threads in the crashing process (not just the thread that caught the
signal), a full memory map, and a list of all open file descriptors.
Before Android 8.0, crashes were handled by the
debuggerd and debuggerd64 daemons. In Android 8.0 and higher,
crash_dump32 and crash_dump64 are spawned as needed.
It's possible for the crash dumper to attach only if nothing else is
already attached, which means that using tools such as strace or
lldb prevent crash dumps from occurring.
Example output (with timestamps and extraneous information removed):
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'Android/aosp_angler/angler:7.1.1/NYC/enh12211018:eng/test-keys'
Revision: '0'
ABI: 'arm'
pid: 17946, tid: 17949, name: crasher >>> crasher <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xc
r0 0000000c r1 00000000 r2 00000000 r3 00000000
r4 00000000 r5 0000000c r6 eccdd920 r7 00000078
r8 0000461a r9 ffc78c19 sl ab209441 fp fffff924
ip ed01b834 sp eccdd800 lr ecfa9a1f pc ecfd693e cpsr 600e0030
backtrace:
#00 pc 0004793e /system/lib/libc.so (pthread_mutex_lock+1)
#01 pc 0001aa1b /system/lib/libc.so (readdir+10)
#02 pc 00001b91 /system/xbin/crasher (readdir_null+20)
#03 pc 0000184b /system/xbin/crasher (do_action+978)
#04 pc 00001459 /system/xbin/crasher (thread_callback+24)
#05 pc 00047317 /system/lib/libc.so (_ZL15__pthread_startPv+22)
#06 pc 0001a7e5 /system/lib/libc.so (__start_thread+34)
Tombstone written to: /data/tombstones/tombstone_06
The last line of output gives the location of the full tombstone on disk.
If you have the unstripped binaries available, you can get a more detailed
unwind with line number information by pasting the stack into
development/scripts/stack:
development/scripts/stack
Tip: For convenience, if you've run lunch,
then stack is already on your $PATH so you don't need to provide the
full path.
Example output (based on the logcat output above):
Reading native crash info from stdin
03-02 23:53:49.477 17951 17951 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
03-02 23:53:49.477 17951 17951 F DEBUG : Build fingerprint: 'Android/aosp_angler/angler:7.1.1/NYC/enh12211018:eng/test-keys'
03-02 23:53:49.477 17951 17951 F DEBUG : Revision: '0'
03-02 23:53:49.477 17951 17951 F DEBUG : ABI: 'arm'
03-02 23:53:49.478 17951 17951 F DEBUG : pid: 17946, tid: 17949, name: crasher >>> crasher <<<
03-02 23:53:49.478 17951 17951 F DEBUG : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xc
03-02 23:53:49.478 17951 17951 F DEBUG : r0 0000000c r1 00000000 r2 00000000 r3 00000000
03-02 23:53:49.478 17951 17951 F DEBUG : r4 00000000 r5 0000000c r6 eccdd920 r7 00000078
03-02 23:53:49.478 17951 17951 F DEBUG : r8 0000461a r9 ffc78c19 sl ab209441 fp fffff924
03-02 23:53:49.478 17951 17951 F DEBUG : ip ed01b834 sp eccdd800 lr ecfa9a1f pc ecfd693e cpsr 600e0030
03-02 23:53:49.491 17951 17951 F DEBUG :
03-02 23:53:49.491 17951 17951 F DEBUG : backtrace:
03-02 23:53:49.492 17951 17951 F DEBUG : #00 pc 0004793e /system/lib/libc.so (pthread_mutex_lock+1)
03-02 23:53:49.492 17951 17951 F DEBUG : #01 pc 0001aa1b /system/lib/libc.so (readdir+10)
03-02 23:53:49.492 17951 17951 F DEBUG : #02 pc 00001b91 /system/xbin/crasher (readdir_null+20)
03-02 23:53:49.492 17951 17951 F DEBUG : #03 pc 0000184b /system/xbin/crasher (do_action+978)
03-02 23:53:49.492 17951 17951 F DEBUG : #04 pc 00001459 /system/xbin/crasher (thread_callback+24)
03-02 23:53:49.492 17951 17951 F DEBUG : #05 pc 00047317 /system/lib/libc.so (_ZL15__pthread_startPv+22)
03-02 23:53:49.492 17951 17951 F DEBUG : #06 pc 0001a7e5 /system/lib/libc.so (__start_thread+34)
03-02 23:53:49.492 17951 17951 F DEBUG : Tombstone written to: /data/tombstones/tombstone_06
Reading symbols from /huge-ssd/aosp-arm64/out/target/product/angler/symbols
Revision: '0'
pid: 17946, tid: 17949, name: crasher >>> crasher <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xc
r0 0000000c r1 00000000 r2 00000000 r3 00000000
r4 00000000 r5 0000000c r6 eccdd920 r7 00000078
r8 0000461a r9 ffc78c19 sl ab209441 fp fffff924
ip ed01b834 sp eccdd800 lr ecfa9a1f pc ecfd693e cpsr 600e0030
Using arm toolchain from: /huge-ssd/aosp-arm64/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin/
Stack Trace:
RELADDR FUNCTION FILE:LINE
0004793e pthread_mutex_lock+2 bionic/libc/bionic/pthread_mutex.cpp:515
v------> ScopedPthreadMutexLocker bionic/libc/private/ScopedPthreadMutexLocker.h:27
0001aa1b readdir+10 bionic/libc/bionic/dirent.cpp:120
00001b91 readdir_null+20 system/core/debuggerd/crasher.cpp:131
0000184b do_action+978 system/core/debuggerd/crasher.cpp:228
00001459 thread_callback+24 system/core/debuggerd/crasher.cpp:90
00047317 __pthread_start(void*)+22 bionic/libc/bionic/pthread_create.cpp:202 (discriminator 1)
0001a7e5 __start_thread+34 bionic/libc/bionic/clone.cpp:46 (discriminator 1)
You can use stack on an entire tombstone. Example:
stack < FS/data/tombstones/tombstone_05
This is useful if you've just unzipped a bug report in the current directory. For more information about diagnosing native crashes and tombstones, see Diagnosing Native Crashes.
Get a stack trace or tombstone from a running process
You can use the debuggerd tool to get a stack dump from a running process.
From the command line, invoke debuggerd using a process ID (PID) to dump a
full tombstone to stdout. To get just the stack for every thread in
the process, include the -b or --backtrace flag.
Understand a complex unwind
When an app crashes, the stack tends to be pretty complex. The following detailed example highlights many of the complexities:
#00 pc 00000000007e6918 /system/priv-app/Velvet/Velvet.apk (offset 0x346b000)
#01 pc 00000000001845cc /system/priv-app/Velvet/Velvet.apk (offset 0x346b000)
#02 pc 00000000001847e4 /system/priv-app/Velvet/Velvet.apk (offset 0x346b000)
#03 pc 00000000001805c0 /system/priv-app/Velvet/Velvet.apk (offset 0x346b000) (Java_com_google_speech_recognizer_AbstractRecognizer_nativeRun+176)
Frames #00–#03 are from native JNI code that was stored uncompressed in the APK to save disk
space rather than being extracted into a separate .so file. The stack unwinder in
Android 9 and higher doesn't need the extracted .so file to cope with this common
Android-specific case.
Frames #00–#02 don't have symbol names because they were stripped by the developer.
Frame #03 shows that where symbols are available, the unwinder uses them.
#04 pc 0000000000117550 /data/dalvik-cache/arm64/system@priv-app@Velvet@Velvet.apk@classes.dex (offset 0x108000) (com.google.speech.recognizer.AbstractRecognizer.nativeRun+160)
Frame #04 is ahead-of-time compiled Java code. The old unwinder would have stopped here, unable to unwind through Java.
#05 pc 0000000000559f88 /system/lib64/libart.so (art_quick_invoke_stub+584)
#06 pc 00000000000ced40 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
#07 pc 0000000000280cf0 /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344)
#08 pc 000000000027acac /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+948)
#09 pc 000000000052abc0 /system/lib64/libart.so (MterpInvokeDirect+296)
#10 pc 000000000054c614 /system/lib64/libart.so (ExecuteMterpImpl+14484)
Frames #05–#10 are from the ART interpreter implementation.
The stack unwinder in releases lower than Android 9 would have shown these frames without the context
of frame #11 explaining what code the interpreter was interpreting. These frames are useful if
you're debugging ART itself. If you're debugging an app, you can ignore them. Some tools, such as
simpleperf, automatically omit these frames.
#11 pc 00000000001992d6 /system/priv-app/Velvet/Velvet.apk (offset 0x26cf000) (com.google.speech.recognizer.AbstractRecognizer.run+18)
Frame #11 is the Java code being interpreted.
#12 pc 00000000002547a8 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.780698333+496)
#13 pc 000000000025a328 /system/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+216)
#14 pc 000000000027ac90 /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+920)
#15 pc 0000000000529880 /system/lib64/libart.so (MterpInvokeVirtual+584)
#16 pc 000000000054c514 /system/lib64/libart.so (ExecuteMterpImpl+14228)
Frames #12–#16 are the interpreter implementation itself.
#17 pc 00000000002454a0 /system/priv-app/Velvet/Velvet.apk (offset 0x1322000) (com.google.android.apps.gsa.speech.e.c.c.call+28)
Frame #17 is the Java code being interpreted. This Java method corresponds to interpreter frames #12–#16.
#18 pc 00000000002547a8 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.780698333+496)
#19 pc 0000000000519fd8 /system/lib64/libart.so (artQuickToInterpreterBridge+1032)
#20 pc 00000000005630fc /system/lib64/libart.so (art_quick_to_interpreter_bridge+92)
Frames #18–#20 are the VM itself, code to transition from compiled Java code to interpreted Java code.
#21 pc 00000000002ce44c /system/framework/arm64/boot.oat (offset 0xdc000) (java.util.concurrent.FutureTask.run+204)
Frame #21 is the compiled Java method that calls the Java method in #17.
#22 pc 0000000000559f88 /system/lib64/libart.so (art_quick_invoke_stub+584)
#23 pc 00000000000ced40 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
#24 pc 0000000000280cf0 /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344)
#25 pc 000000000027acac /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+948)
#26 pc 0000000000529880 /system/lib64/libart.so (MterpInvokeVirtual+584)
#27 pc 000000000054c514 /system/lib64/libart.so (ExecuteMterpImpl+14228)
Frames #22–#27 are the interpreter implementation, making a method invocation from interpreted code to a compiled method.
#28 pc 00000000003ed69e /system/priv-app/Velvet/Velvet.apk (com.google.android.apps.gsa.shared.util.concurrent.b.e.run+22)
Frame #28 is the Java code being interpreted.
#29 pc 00000000002547a8 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.780698333+496)
#30 pc 0000000000519fd8 /system/lib64/libart.so (artQuickToInterpreterBridge+1032)
#31 pc 00000000005630fc /system/lib64/libart.so (art_quick_to_interpreter_bridge+92)
Frames #29–#31 are another transition between compiled code and interpreted code.
#32 pc 0000000000329284 /system/framework/arm64/boot.oat (offset 0xdc000) (java.util.concurrent.ThreadPoolExecutor.runWorker+996)
#33 pc 00000000003262a0 /system/framework/arm64/boot.oat (offset 0xdc000) (java.util.concurrent.ThreadPoolExecutor$Worker.run+64)
#34 pc 00000000002037e8 /system/framework/arm64/boot.oat (offset 0xdc000) (java.lang.Thread.run+72)
Frames #32–#34 are compiled Java frames calling each other directly. In this case the native call stack is the same as the Java call stack.
#35 pc 0000000000559f88 /system/lib64/libart.so (art_quick_invoke_stub+584)
#36 pc 00000000000ced40 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
#37 pc 0000000000280cf0 /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344)
#38 pc 000000000027acac /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+948)
#39 pc 0000000000529f10 /system/lib64/libart.so (MterpInvokeSuper+1408)
#40 pc 000000000054c594 /system/lib64/libart.so (ExecuteMterpImpl+14356)
Frames #35–#40 are the interpreter itself.
#41 pc 00000000003ed8e0 /system/priv-app/Velvet/Velvet.apk (com.google.android.apps.gsa.shared.util.concurrent.b.i.run+20)
Frame #41 is the Java code being interpreted.
#42 pc 00000000002547a8 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.780698333+496)
#43 pc 0000000000519fd8 /system/lib64/libart.so (artQuickToInterpreterBridge+1032)
#44 pc 00000000005630fc /system/lib64/libart.so (art_quick_to_interpreter_bridge+92)
#45 pc 0000000000559f88 /system/lib64/libart.so (art_quick_invoke_stub+584)
#46 pc 00000000000ced40 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
#47 pc 0000000000460d18 /system/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
#48 pc 0000000000461de0 /system/lib64/libart.so (art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue*)+424)
#49 pc 000000000048ccb0 /system/lib64/libart.so (art::Thread::CreateCallback(void*)+1120)
Frames #42–#49 are the VM itself. This time it's the code that starts running Java on a new thread.
#50 pc 0000000000082e24 /system/lib64/libc.so (__pthread_start(void*)+36)
#51 pc 00000000000233bc /system/lib64/libc.so (__start_thread+68)
Frames #50–#51 are how all threads should start. This is the libc
new thread start code.