mlc-ai / mlc-ai/tokenizers-cpp
AddressSanitizer: heap-use-after-free on addres xxx
Open
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 512
- Forks
- 132
- PR merge metrics
- No merged PRs in 30d
Description
Asan messages:
READ of size 173 at 0x00766930acc0 thread T7
#0 0x76a133ab38 in __interceptor_memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:827
#1 0x76a0e1a544 in tokenizers::HFTokenizer::Decode[abi:cxx11](std::vector<int, std::allocator<int> > const&)
0x00766930acc0 is located 0 bytes inside of 173-byte region [0x00766930acc0,0x00766930ad6d)
freed by thread T0 here:
#0 0x76a13a9fe8 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:127
#1 0x76a0e1d654 in tokenizers_decode
previously allocated by thread T7 here:
#0 0x76a13aa2f4 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
#1 0x76a0e2abcc in alloc::str::join_generic_copy::h968362ed642e72e2
My solution:
TokenizerObj.decode() must be called in one thread.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at TokenizerObj.decode() and the tokenizers_decode entry point, then compare them with the HFTokenizer::Decode stack frame in the ASan report. Reproduce the heap-use-after-free under concurrent decoding and determine the ownership or threading boundary involved; done means the reproducer no longer reports the ASan failure without requiring callers to serialize decode calls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100