Understand HWASan reports

When the HWASan tool detects a memory bug, the process is terminated with abort(), and a report is printed to stderr and logcat. Like all native crashes on Android, HWASan errors are under /data/tombstones.

Example report

Compared to regular native crashes, HWASan carries extra information in the Abort message field near the top of the tombstone. Here is a sample heap-based crash. For stack bugs, see the note for the stack-specific sections.

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'google/flame_hwasan/flame:Tiramisu/MASTER/7956676:userdebug/dev-keys'
Revision: 'DVT1.0'
ABI: 'arm64'
Timestamp: 2019-04-24 01:13:22+0000
pid: 11154, tid: 11154, name: sensors@1.0-ser  >>> /vendor/bin/hw/android.hardware.sensors@1.0-service <<<
signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
Abort message: '

[...]

[0x00433ae20040,0x00433ae20060) is a small unallocated heap chunk; size: 32 offset: 5








[ … regular crash dump follows …]

This is similar to an AddressSanitizer report. Unlike those, almost all HWASan bugs are tag-mismatch errors, that is, a memory access where a pointer tag doesn't match the corresponding memory tag. This could be any of the following:

  • Out of bounds access on stack or heap
  • Use-after-free error on heap
  • Use-after-return error on stack

Sections

Here is an explanation of each of the sections of the HWASan report.

Access error

Contains information about the bad memory access, including:

  • Access type (READ versus WRITE)
  • Access size (how many bytes were attempted to be accessed)
  • Thread number of the access
  • Pointer and memory tags (for advanced debugging)

Access stack trace

Stack trace of the bad memory access. See Symbolization to symbolize.

Cause

The potential cause for the bad access. If there are multiple candidates, they're listed in order of descending likelihood. Precedes the detailed info about the potential cause. HWASan can diagnose the following causes:

  • Use after free
  • Stack tag mismatch, which can be stack use after return, stack use after scope, or out of bounds
  • Heap buffer overflow
  • Global overflow

Memory information

Describes what HWASan knows about the memory being accessed, and can differ based on the bug type:

Bug type Cause Report format
Tag mismatch Use after free Use this report format:
<address> is located N bytes inside of M-byte region [<start>, <end>)
freed by thread T0 here:
Heap buffer overflow Note that this can also be an underflow.
<address> is located N bytes to the right of M-byte region [<start>, <end>)
allocated here:
Stack tag mismatch Stack reports don't differentiate between overflow or underflow and use-after-return bugs. In addition, to find the stack allocation that is the source of the error, an offline symbolization step is required. See Understand stack reports.
Invalid free Use after free A double free bug. If this happens on process shutdown, this can signify an ODR violation. 
<address> is located N bytes inside of M-byte region [<start>, <end>)
freed by thread T0 here:
Can't describe address Either a wild free (free of memory that hadn't been allocated before), or a double free after the allocated memory was evicted from HWASan's free buffer.
0x... is HWAsan shadow memory A wild free, as the app was attempting to free memory that is internal to HWASan.

Deallocation stack trace

Stack trace of where the memory was deallocated. Present only for use-after-free or invalid-free bugs. See Symbolization to symbolize.

Allocation stack trace

Stack trace of where the memory was allocated. See Symbolization to symbolize.

Advanced debugging information

The HWASan report also features some advanced debugging information, including (in order):

  1. The list of threads in the process
  2. The list of threads in the process
  3. The value of the memory tags near the faulting memory
  4. The dump of the registers at the point of memory access

Memory tag dump

You can use the tag memory dump to look for nearby memory allocations with the same tag as the pointer tag. These tags can point to an out of bounds access with a large offset. One tag corresponds to 16 bytes of memory; the pointer tag is the top 8 bits of the address. The tag memory dump can give hints, for example, the following is a buffer overflow to the right:

tags: ad/5c (ptr/mem)
[...]
Memory tags around the buggy address (one tag corresponds to 16 bytes):
  0x006f33ae1ff0: 0e  0e  0e  57  20  20  20  20  20  2e  5e  5e  5e  5e  5e  b5
=>0x006f33ae2000: f6  f6  f6  f6  f6  4c  ad  ad  ad  ad  ad  ad [5c] 5c  5c  5c
  0x006f33ae2010: 5c  04  2e  2e  2e  2e  2e  2f  66  66  66  66  66  80  6a  6a
Tags for short granules around the buggy address (one tag corresponds to 16 bytes):
  0x006f33ae1ff0: ab  52  eb  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..
=>0x006f33ae2000: ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  .. [..] ..  ..  ..
  0x006f33ae2010: ..  5c  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..  ..

Note the run of 6 × 16 = 96 bytes of ad tags to the left that match the pointer tag.

If the size of an allocation isn't a multiple of 16, the remainder of the size is stored as the memory tag and the tag is stored as a short granule tag. In the previous example, just after the bolded allocation tagged ad, we have a 5 × 16 + 4 = 84 byte allocation of tag 5c.

A zero memory tag (for example, tags: ad/00 (ptr/mem)) indicates a stack-use-after-return bug.

Register dump

The register dump in HWASan reports corresponds to the instruction that performed the invalid memory access. This dump is followed by another register dump from the regular Android signal handler. Ignore the second dump, as it was taken when HWASan called abort() and isn't relevant to the bug.

Symbolization

To get function names and line numbers in stack traces (and get variable names for use-after-scope bugs), an offline symbolization step is needed.

First-time setup: install llvm-symbolizer

To symbolize, your system must have llvm-symbolizer installed and accessible from $PATH. On Debian, you can install it using sudo apt install llvm.

Obtain symbol files

For symbolization, we require unstripped binaries containing symbols. Their location depends on the type of build:

  • For local builds, the symbol files are in out/target/product/<product>/symbols/.
  • For AOSP builds (for example, flashed from Android Flash Tool), the builds are on Android CI. In the Artifacts for the build, there is a ${PRODUCT}-symbols-${BUILDID}.zip file.
  • For internal builds from your organization, check your organization's documentation for help obtaining symbol files.

Symbolize

hwasan_symbolize --symbols <DECOMPRESSED_DIR>/out/target/product/*/symbols < crash

Understand stack reports

For bugs that occur with stack variables, the HWASan report contains details like this:

Cause: stack tag-mismatch
Address 0x007d4d251e80 is located in stack of thread T64
Thread: T64 0x0074000b2000 stack: [0x007d4d14c000,0x007d4d255cb0) sz: 1088688 tls: [0x007d4d255fc0,0x007d4d259000)
Previously allocated frames:
  record_addr:0x7df7300c98 record:0x51ef007df3f70fb0  (/apex/com.android.art/lib64/libart.so+0x570fb0)
  record_addr:0x7df7300c90 record:0x5200007df3cdab74  (/apex/com.android.art/lib64/libart.so+0x2dab74)
  [...]

To help you understand stack bugs, HWASan keeps track of past stack frames. HWASan doesn't transform these into human-understandable content in the bug report, and requires an additional symbolization step.

ODR violations

Some use-after-free bugs reported by HWASan can indicate a One Definition Rule (ODR) violation. An ODR violation happens when the same variable is defined multiple times in the same program. This also means that the variable is destructed multiple times, which can lead to the use-after-free error.

After symbolization, ODR violations show a use-after-free error with __cxa_finalize, on both the invalid access stack and the freed here stack. The previously allocated here stack contains __dl__ZN6soinfo17call_constructorsEv and should point at the location in your program that defines the variable higher on the stack.

The ODR can be violated if static libraries are used. If a static library that defines a C++ global is linked into multiple shared libraries or executables, multiple definitions of the same symbol might exist in the same address space, which causes an ODR error.

Troubleshooting

This section describes some errors and how to address them.

HWAddressSanitizer can not describe address in more detail

Sometimes HWASan can run out of space for information about past memory allocations. In that case, the report contains only one stack trace for the immediate memory access, followed by a note:

HWAddressSanitizer can not describe address in more detail.

In some cases, you can resolve this by running the test multiple times. Another option is to increase HWASan history size. You can do this globally in build/soong/cc/sanitize.go (look for hwasanGlobalOptions), or in your process environment (try adb shell echo $HWASAN_OPTIONS to see the current settings).

This error can also happen if the accessed memory isn't mapped, or allocated by a non-HWASan aware allocator. In this case, the mem tag listed in the crash header is generally 00. If you have access to the full tombstone, it might be helpful to consult the memory maps dump to find out which mapping (if any) the address belongs to.

Nested bug in the same thread

This means there was a bug while generating the HWASan crash-report. This is usually due to a bug in the HWASan runtime. File a bug and provide instructions for how to reproduce the issue if possible.