Understanding HWASan reports

When the HWASan tool detects a memory bug, the process is terminated with abort(), and a report is printed to stderr and logcat. Like all native crashes on Android, HWASan errors can be found under /data/tombstones.

Compared to regular native crashes, HWASan carries extra information in the “Abort message” field near the top of the tombstone. See a sample heap-based crash below (for stack bugs, see the note below for the stack-specific sections).

Example report

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'google/flame_hwasan/flame:Tiramisu/MASTER/7956676:userdebug/dev-keys'
Revision: 'DVT1.0'
ABI: 'arm64'
Timestamp: 2019-04-24 01:13:22+0000
pid: 11154, tid: 11154, name: sensors@1.0-ser  >>> /vendor/bin/hw/android.hardware.sensors@1.0-service <<<
signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
Abort message: '

[...]

[0x00433ae20040,0x00433ae20060) is a small unallocated heap chunk; size: 32 offset: 5








[ … regular crash dump follows …]

This is very similar to an AddressSanitizer report. Unlike those, almost all HWASan bugs are “tag-mismatch”, i.e. a memory access where a pointer tag does not match the corresponding memory tag. This could be one of

  • out of bounds access on stack or heap
  • use after free on heap
  • use after return on stack

Sections

An explanation of each of the sections of the HWASan report is below:

Access Error

Contains information about the bad memory access, including:

  • Access Type ("READ" vs. "WRITE")
  • Access size (how many bytes were attempted to be accessed)
  • Thread number of the access
  • Pointer and memory tags (for advanced debugging)

Access Stack Trace

Stack trace of the bad memory access. See the Symbolization section to symbolize.

Cause

The potential cause for the bad access. If there are multiple candidates, they are listed in order of descending likelihood. Precedes the detailed info about the potential cause. HWASan can diagnose the following causes:

  • use-after-free
  • stack tag-mismatch: this can be stack use-after-return / use after-scope, or out of bounds
  • heap-buffer-overflow
  • global-overflow

Memory Information

Describes what HWASan knows about the memory being accessed, and may differ based on the bug type.

Bug Type Cause Report Format
tag-mismatch use-after-free
<address> is located N bytes inside of M-byte region [<start>, <end>)
  freed by thread T0 here:
heap-buffer-overflow Note that this can also be an underflow.
<address> is located N bytes to the right of M-byte region [<start>, <end>)
  allocated here:
stack tag-mismatch Stack reports do not differentiate between overflow/underflow and use-after-return bugs. In addition, to find the stack allocation that is the source of the error, an offline symbolization step is required. See the Understanding stack reports section below.
invalid-free use-after-free This is a double free bug. If this happens on process shutdown, this can signify an ODR violation.
<address> is located N bytes inside of M-byte region [<start>, <end>)
  freed by thread T0 here:
cannot describe address Either a wild free (free of memory that hadn't been allocated before), or a double free after the allocated memory was evicted from HWASan's free buffer.
0x… is HWAsan shadow memory. This is definitely a wild free, as the application was attempting to free memory that is internal to HWASan.

Deallocation Stack Trace

Stack trace of where the memory was deallocated. Only present for use-after-free or invalid-free bugs. See Symbolization section to symbolize.

Allocation Stack Trace

Stack trace of where the memory was allocated. See Symbolization section to symbolize.

Advanced Debugging Information

The HWASan report also features some advanced debugging information, including (in order):

  1. The list of threads in the process
  2. The list of threads in the process
  3. The value of the memory tags near the faulting memory
  4. The dump of the registers at the point of memory access

Memory Tag Dump

The tag memory dump can be used to look for nearby memory allocations with the same tag as the pointer tag. These could point to an out of bounds access with a large offset. One tag corresponds to 16 bytes of memory; the pointer tag is the top 8 bits of the address. The tag memory dump can give hints, for example the following is a buffer overflow to the right:

tags: ad/5c (ptr/mem)
[...]
Memory tags around the buggy address (one tag corresponds to 16 bytes):
  0x006f33ae1ff0: 0e  0e  0e  57  20  20  20  20  20  2e  5e  5e  5e  5e  5e  b5
=>0x006f33ae2000: f6  f6  f6  f6  f6  4c  ad  ad  ad  ad  ad  ad [5c] 5c  5c  5c
  0x006f33ae2010: 5c  04  2e  2e  2e  2e  2e  2f  66  66  66  66  66  80  6a  6a
Tags for short granules around the buggy address (one tag corresponds to 16 bytes):
  0x006f33ae1ff0: ab  52  eb  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..
=>0x006f33ae2000: ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  .. [..] ..  ..  ..
  0x006f33ae2010: ..  5c  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..
(note the run of 6 × 16 = 96 bytes of “ad” tags to the left that match the pointer tag).

If the size of an allocation is not a multiple of 16, the remainder of the size will be stored as the memory tag and the tag will be stored as a short granule tag. In the example above just after the bolded allocation tagged ad, we have a 5 × 16 + 4 = 84 byte allocation of tag 5c.

A zero memory tag (ex. tags: ad/00 (ptr/mem)) usually indicates a stack-use-after-return bug.

Register Dump

The register dump in HWASan reports corresponds to the actual instruction that performed the invalid memory access. It is followed by another register dump from the regular Android signal handler - ignore the second one, it is taken when HWASan called abort() and is not relevant to the bug.

Symbolization

To get function names and line numbers in stack traces (and get variable names for use-after-scope bugs), an offline symbolization step is needed.

First-time setup: install llvm-symbolizer

To symbolize, your system must have llvm-symbolizer installed and accessible from $PATH. On Debian, you can install it using sudo apt install llvm.

Obtain symbol files

For symbolization, we require unstripped binaries containing symbols. Where these can be found depends on the type of build:

For local builds, the symbol files can be found in out/target/product/<product>/symbols/.

For AOSP builds (e.g. flashed from Flashstation), the builds can be found on Android CI. In the "Artifacts" for the build, there will be a `${PRODUCT}-symbols-${BUILDID}.zip` file.

For internal builds from your organization, check your organization's documentation for help obtaining symbol files.

Symbolize

hwasan_symbolize –-symbols <DECOMPRESSED_DIR>/out/target/product/*/symbols < crash

Understanding Stack Reports

For bugs that occur with stack variables, the HWASan report will contain details like this:

Cause: stack tag-mismatch
Address 0x007d4d251e80 is located in stack of thread T64
Thread: T64 0x0074000b2000 stack: [0x007d4d14c000,0x007d4d255cb0) sz: 1088688 tls: [0x007d4d255fc0,0x007d4d259000)
Previously allocated frames:
  record_addr:0x7df7300c98 record:0x51ef007df3f70fb0  (/apex/com.android.art/lib64/libart.so+0x570fb0)
  record_addr:0x7df7300c90 record:0x5200007df3cdab74  (/apex/com.android.art/lib64/libart.so+0x2dab74)
  [...]
	

To allow stack bugs to be understood, HWASan keeps track of the stack frames that happened in the past. Currently, HWASan doesn't transform this into human-understandable content in the bug report, and requires an additional symbolization step.

ODR Violations

Some use-after-free bugs reported by HWASan can also indicate an One Definition Rule (ODR) violation. An ODR violation happens when the same variable is defined multiple times in the same program. This also means that the variable is destructed multiple times, which may lead to the use-after-free error.

After symbolization, ODR violations show a use-after-free with __cxa_finalize, on both the invalid access stack and the "freed here" stack. The "previously allocated here" stack contains __dl__ZN6soinfo17call_constructorsEv and should point at the location in your program that defines the variable higher on the stack.

One reason why the ODR may be violated is when static libraries are used. If a static library that defines a C++ global is linked into multiple shared libraries or executables, multiple definitions of the same symbol may end up existing in the same address space, which will cause an ODR error.

Troubleshooting

"HWAddressSanitizer can not describe address in more detail."

Sometimes HWASan can run out of space for information about past memory allocations. In that case, the report will contain only one stack trace for the immediate memory access, followed by a note:

  HWAddressSanitizer can not describe address in more detail.

In some cases this can be resolved by running the test multiple times. Another option is to increase HWASan history size. This can be done globally in build/soong/cc/sanitize.go (look for hwasanGlobalOptions), or in your process environment (try adb shell echo $HWASAN_OPTIONS to see the current settings).

This can also happen if the accessed memory is not mapped, or allocated by a non-HWASan aware allocator. In this case, the mem tag listed in the crash header will generally be 00. If you have access to the full tombstone, it might be helpful to consult the memory maps dump to find out which mapping (if any) the address belongs to.

"nested bug in the same thread"

This means there was a bug while generating the HWASan crash-report. This is usually due to a bug in the HWASan runtime, please file a bug and provide instructions on how to reproduce the issue if possible.