Possible use-after-unmap in buildSmallBlockMatcherProto during hs_compile_multi
- Dominant language
- C++
- Stars
- 5.5k
- Forks
- 816
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 2
Description
### Summary
We are seeing a deterministic crash inside Hyperscan 5.4.2 while calling
`hs_compile_multi()` with 2787 expressions.
The crash occurs later in `buildIncludedIdMap()` while reading the first
`hwlmLiteral` from `fproto->hwlmProto->lits`. Kernel-side munmap tracing
shows that the backing address range was previously released from
`buildSmallBlockMatcherProto()` through the ASan allocator.
This appears to be a possible ownership/lifetime issue involving a temporary
MatcherProto and the final LitProto/HWLMProto.
### Environment
- Hyperscan: 5.4.2
- Architecture: x86_64 Linux
- Application compiler: Clang 20.1.8
- libstdc++ headers/runtime: GCC 14.3.0
- Application build: optimized release build with AddressSanitizer
- Application ASan flags: `-fsanitize=address`
- jemalloc disabled
- `libhs.so.5` itself is not ASan-instrumented
- `libhs.so.5` uses the normal dynamic C/C++ allocators:
`malloc`, `free`, `operator new`, and `operator delete`
The application calls `hs_compile_multi()` synchronously. The expression,
flag, ID, platform, output database, and error-pointer arrays remain valid
for the duration of the call.
### Crash stack
The relevant stack is:
```text
#10 std::unordered_map<...>::find(...)
#11 ue2::contains(...)
src/util/container.h:51
#12 ue2::buildIncludedIdMap(...)
src/rose/rose_build_bytecode.cpp:2885
#13 ue2::findInclusionGroups(...)
src/rose/rose_build_bytecode.cpp:2910
#14 ue2::buildLiteralPrograms(...)
src/rose/rose_build_bytecode.cpp:3061
#15 ue2::RoseBuildImpl::buildFinalEngine(...)
src/rose/rose_build_bytecode.cpp:3715
#16 ue2::RoseBuildImpl::buildRose(...)
#17 ue2::generateRoseEngine(...)
#18 ue2::build(...)
#19 ue2::hs_compile_multi_int(...)
src/hs.cpp:255
#20 hs_compile_multi(...)
src/hs.cpp:438
At the fault:
fproto = 0x7bc6eb356ea0
drproto = NULL
eproto = NULL
sbproto = NULL
elements = 2787
&lit.id = 0x7b96e49bc820
GDB cannot read lit.id because its mapping has already been removed.
### munmap evidence
We added temporary tracing around the kernel munmap syscall and captured
the successful munmap that covers the fault address:
tgid=9749 tid=9749
addr=0x7b96e4978000
len=0x6a000
end=0x7b96e49e2000
The fault address lies inside this range:
0x7b96e49bc820 - 0x7b96e4978000 = 0x44820
The captured userspace stack associated with that munmap resolves to:
__sanitizer::internal_munmap(void *, unsigned long) + 7
__sanitizer::UnmapOrDie(void *, unsigned long, bool) + 32
__asan::asan_free(...) + 471
___interceptor_free.part.0
ue2::buildSmallBlockMatcherProto(...) + 434
ue2::RoseBuildImpl::buildFinalEngine(...) + 6036
ue2::RoseBuildImpl::buildRose(...)
ue2::generateRoseEngine(...)
ue2::build(...)
ue2::hs_compile_multi_int(...)
hs_compile_multi(...)
Therefore, the mapping containing lit.id was released through a normal
C++/ASan free path originating from buildSmallBlockMatcherProto().
### Reproducibility
We have observed this multiple times with different ASLR layouts.
For example, in another run:
munmap start = 0x7be5c02e9000
munmap length = 0x6a000
fault address = 0x7be5c032d820
Again:
fault address - munmap start = 0x44820
The following properties are stable across runs:
- 2787 expressions
- crash at buildIncludedIdMap() line 2885
- release originates from buildSmallBlockMatcherProto() + 434
- munmap length is 0x6a000
- fault offset inside the unmapped allocation is 0x44820
This makes random address corruption less likely.
### Additional observations
The userspace stack memory also contains addresses resolving to:
std::__introsort_loop<... ue2::AccelString ...>
__interceptor_memcmp
These are not necessarily active unwind frames, but they indicate that the
same function previously processed/sorted a temporary AccelString vector.
Disassembly around buildSmallBlockMatcherProto() suggests that the release
may occur during cleanup of a local MatcherProto, possibly around an early
return or a branch handling a MatcherProto with one literal.
Our current hypothesis is:
1. buildSmallBlockMatcherProto() creates a temporary MatcherProto.
2. Some vector storage is copied/moved or otherwise referenced by the final
LitProto/HWLMProto.
3. Cleanup of the temporary object frees the storage.
4. buildIncludedIdMap() later iterates fproto->hwlmProto->lits, whose
storage still points into the freed mapping.
This is a hypothesis, not yet a confirmed source-level root cause.
### Questions
1. Is there any known ownership or move/copy issue between MatcherProto,
LitProto, and HWLMProto in buildSmallBlockMatcherProto()?
2. Is the early-return/single-literal path expected to destroy storage that
may already have been transferred to the resulting LitProto?
3. Are there known compatibility issues when a non-ASan Hyperscan shared
library is loaded into an ASan-instrumented executable?
4. Would you recommend any specific internal assertions or address logging
to verify ownership of mp.lits, mp.accel, fproto->accel_lits, and
fproto->hwlmProto->lits?
We can provide the exact expression set privately if needed.
Contributor guide
Research direction
Start with src/rose/rose_build_bytecode.cpp at buildSmallBlockMatcherProto(), buildIncludedIdMap(), and the surrounding buildFinalEngine() path, then reproduce hs_compile_multi() with the 2787-expression case under ASan. Trace ownership and destruction of MatcherProto, LitProto, HWLMProto, mp.lits, mp.accel, fproto->accel_lits, and fproto->hwlmProto->lits. Done means the lifetime cause is confirmed and the reproducer no longer accesses unmapped storage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100