intel / intel/hyperscan

Possible use-after-unmap in buildSmallBlockMatcherProto during hs_compile_multi

Open
#477 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
5.5k
Forks
816
Avg merge
4d 21h
Merged PRs (30d)
2

Description

### Summary

We are seeing a deterministic crash inside Hyperscan 5.4.2 while calling
`hs_compile_multi()` with 2787 expressions.

The crash occurs later in `buildIncludedIdMap()` while reading the first
`hwlmLiteral` from `fproto->hwlmProto->lits`. Kernel-side munmap tracing
shows that the backing address range was previously released from
`buildSmallBlockMatcherProto()` through the ASan allocator.

This appears to be a possible ownership/lifetime issue involving a temporary
MatcherProto and the final LitProto/HWLMProto.

### Environment

- Hyperscan: 5.4.2
- Architecture: x86_64 Linux
- Application compiler: Clang 20.1.8
- libstdc++ headers/runtime: GCC 14.3.0
- Application build: optimized release build with AddressSanitizer
- Application ASan flags: `-fsanitize=address`
- jemalloc disabled
- `libhs.so.5` itself is not ASan-instrumented
- `libhs.so.5` uses the normal dynamic C/C++ allocators:
`malloc`, `free`, `operator new`, and `operator delete`

The application calls `hs_compile_multi()` synchronously. The expression,
flag, ID, platform, output database, and error-pointer arrays remain valid
for the duration of the call.

### Crash stack

The relevant stack is:

```text
#10 std::unordered_map<...>::find(...)
#11 ue2::contains(...)
src/util/container.h:51
#12 ue2::buildIncludedIdMap(...)
src/rose/rose_build_bytecode.cpp:2885
#13 ue2::findInclusionGroups(...)
src/rose/rose_build_bytecode.cpp:2910
#14 ue2::buildLiteralPrograms(...)
src/rose/rose_build_bytecode.cpp:3061
#15 ue2::RoseBuildImpl::buildFinalEngine(...)
src/rose/rose_build_bytecode.cpp:3715
#16 ue2::RoseBuildImpl::buildRose(...)
#17 ue2::generateRoseEngine(...)
#18 ue2::build(...)
#19 ue2::hs_compile_multi_int(...)
src/hs.cpp:255
#20 hs_compile_multi(...)
src/hs.cpp:438

At the fault:

fproto = 0x7bc6eb356ea0
drproto = NULL
eproto = NULL
sbproto = NULL
elements = 2787

&lit.id = 0x7b96e49bc820

GDB cannot read lit.id because its mapping has already been removed.

### munmap evidence

We added temporary tracing around the kernel munmap syscall and captured
the successful munmap that covers the fault address:

tgid=9749 tid=9749
addr=0x7b96e4978000
len=0x6a000
end=0x7b96e49e2000

The fault address lies inside this range:

0x7b96e49bc820 - 0x7b96e4978000 = 0x44820

The captured userspace stack associated with that munmap resolves to:

__sanitizer::internal_munmap(void *, unsigned long) + 7
__sanitizer::UnmapOrDie(void *, unsigned long, bool) + 32
__asan::asan_free(...) + 471
___interceptor_free.part.0
ue2::buildSmallBlockMatcherProto(...) + 434
ue2::RoseBuildImpl::buildFinalEngine(...) + 6036
ue2::RoseBuildImpl::buildRose(...)
ue2::generateRoseEngine(...)
ue2::build(...)
ue2::hs_compile_multi_int(...)
hs_compile_multi(...)

Therefore, the mapping containing lit.id was released through a normal
C++/ASan free path originating from buildSmallBlockMatcherProto().

### Reproducibility

We have observed this multiple times with different ASLR layouts.

For example, in another run:

munmap start = 0x7be5c02e9000
munmap length = 0x6a000
fault address = 0x7be5c032d820

Again:

fault address - munmap start = 0x44820

The following properties are stable across runs:

- 2787 expressions
- crash at buildIncludedIdMap() line 2885
- release originates from buildSmallBlockMatcherProto() + 434
- munmap length is 0x6a000
- fault offset inside the unmapped allocation is 0x44820

This makes random address corruption less likely.

### Additional observations

The userspace stack memory also contains addresses resolving to:

std::__introsort_loop<... ue2::AccelString ...>
__interceptor_memcmp

These are not necessarily active unwind frames, but they indicate that the
same function previously processed/sorted a temporary AccelString vector.

Disassembly around buildSmallBlockMatcherProto() suggests that the release
may occur during cleanup of a local MatcherProto, possibly around an early
return or a branch handling a MatcherProto with one literal.

Our current hypothesis is:

1. buildSmallBlockMatcherProto() creates a temporary MatcherProto.
2. Some vector storage is copied/moved or otherwise referenced by the final
LitProto/HWLMProto.

3. Cleanup of the temporary object frees the storage.
4. buildIncludedIdMap() later iterates fproto->hwlmProto->lits, whose
storage still points into the freed mapping.

This is a hypothesis, not yet a confirmed source-level root cause.

### Questions

1. Is there any known ownership or move/copy issue between MatcherProto,
LitProto, and HWLMProto in buildSmallBlockMatcherProto()?

2. Is the early-return/single-literal path expected to destroy storage that
may already have been transferred to the resulting LitProto?

3. Are there known compatibility issues when a non-ASan Hyperscan shared
library is loaded into an ASan-instrumented executable?

4. Would you recommend any specific internal assertions or address logging
to verify ownership of mp.lits, mp.accel, fproto->accel_lits, and
fproto->hwlmProto->lits?

We can provide the exact expression set privately if needed.

Contributor guide

Open the contributing guide

Research direction

Start with src/rose/rose_build_bytecode.cpp at buildSmallBlockMatcherProto(), buildIncludedIdMap(), and the surrounding buildFinalEngine() path, then reproduce hs_compile_multi() with the 2787-expression case under ASan. Trace ownership and destruction of MatcherProto, LitProto, HWLMProto, mp.lits, mp.accel, fproto->accel_lits, and fproto->hwlmProto->lits. Done means the lifetime cause is confirmed and the reproducer no longer accesses unmapped storage.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.