[llvm-cov] `llvm-cov report` intermittently crashes with `-num-threads > 1`
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
`llvm-cov-20 report` intermittently crashes when generating a summary from multiple instrumented executables. The crash only appears when more than one worker thread is used. `-num-threads=1` seems to be a reliable workaround.
## Environment
I have currently tested this behavior on Ubuntu 24.04 with LLVM 20.1.8 and 22.1.8.
## Reproducer
The attached `reproducer` directory contains:
- `Overall.profdata`
- four instrumented test executables with debug information
- `crash.sh`, which repeatedly runs `llvm-cov-20 report`
Run:
```sh
cd reproducer
./crash.sh
```
The minimized command is:
```sh
llvm-cov-22 report \
ut_Date_Time \
-object ut_Unit_Conversion \
-object ut_Isolette \
-object ut_Room_Air \
-instr-profile=Overall.profdata \
-dump \
-num-threads=8 \
> /dev/null
```
The crash is nondeterministic. On my machine it usually happens within a few iterations with `-num-threads=$(nproc)`.
## Expected behavior
`llvm-cov report` should either complete successfully or report a normal diagnostic. It should not crash or corrupt heap state.
## Actual behavior
The failure is nondeterministic and the exact symptom varies. I have observed heap corruption diagnostics, segmentation faults, aborts, and in one run the LLVM crash-report message was printed but the process did not exit at all.
A few representative crashes are shown below.
Example 1
```plain
+ llvm-cov-22 report ut_Date_Time -object ut_Unit_Conversion -object ut_Isolette -object ut_Room_Air -instr-profile=Overall.profdata -dump -num-threads=4
double free or corruption (!prev)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
#0 0x00007588094dd17d llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xcdd17d)
#1 0x00007588094da120 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
#2 0x0000758808045330 (/lib/x86_64-linux-gnu/libc.so.6+0x45330)
#3 0x000075880809eb2c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
#4 0x000075880809eb2c __pthread_kill_internal ./nptl/pthread_kill.c:78:10
#5 0x000075880809eb2c pthread_kill ./nptl/pthread_kill.c:89:10
#6 0x000075880804527e raise ./signal/../sysdeps/posix/raise.c:27:6
#7 0x00007588080288ff abort ./stdlib/abort.c:81:7
#8 0x00007588080297b6 _IO_acquire_lock_fct ./libio/libioP.h:979:6
#9 0x00007588080297b6 _IO_peekc_locked ./libio/peekc.c:37:3
#10 0x00007588080a8ff5 ./malloc/malloc.c:5775:3
#11 0x00007588080ab12c _int_free_merge_chunk ./malloc/malloc.c:4679:5
#12 0x00007588080ab529 _int_free ./malloc/malloc.c:4648:5
#13 0x00007588080addce cfree ./malloc/malloc.c:3401:3
#14 0x00007588094adae7 llvm::raw_ostream::SetBuffered() (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xcadae7)
#15 0x00007588094ae537 llvm::raw_ostream::write(char const*, unsigned long) (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xcae537)
#16 0x000058ab3f596306 llvm::CoverageReport::prepareSingleFileReport(llvm::StringRef, llvm::coverage::CoverageMapping const*, llvm::CoverageViewOptions const&, unsigned int, llvm::FileCoverageSummary*, llvm::CoverageFilter const*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x48306)
#17 0x000058ab3f594f1d void llvm::detail::UniqueFunctionBase::CallImpl>(void*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x46f1d)
#18 0x000058ab3f5652e7 std::_Function_handler (), std::__future_base::_Task_setter, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker>>, void>>::_M_invoke(std::_Any_data const&) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x172e7)
#19 0x000058ab3f567249 std::__future_base::_State_baseV2::_M_do_set(std::function ()>*, bool*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x19249)
#20 0x00007588080a1ed3 __pthread_once_slow ./nptl/pthread_once.c:118:7
#21 0x000058ab3f56719c std::__future_base::_Deferred_state>>, void>::_M_complete_async() (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x1919c)
#22 0x000058ab3f5664ec void llvm::detail::UniqueFunctionBase::CallImpl llvm::ThreadPoolInterface::asyncImpl(llvm::unique_function, llvm::ThreadPoolTaskGroup*)::'lambda'()>(void*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x184ec)
#23 0x000075880944d639 llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xc4d639)
#24 0x000075880944e2e7 void* llvm::thread::ThreadProxy>(void*) ThreadPool.cpp:0:0
#25 0x000075880809caa4 start_thread ./nptl/pthread_create.c:447:8
#26 0x0000758808129c6c clone3 ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:80:0
./crash.sh: line 20: 214489 Aborted llvm-cov-22 report "${overall_objects[@]}" -instr-profile=Overall.profdata -dump "-num-threads=4" > /dev/null
```
Example 2
```plain
double free or corruption (out)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
malloc(): unaligned tcache chunk detected
./crash.sh: line 20: 217872 Aborted llvm-cov-22 report "${overall_objects[@]}" -instr-profile=Overall.profdata -dump "-num-threads=4" > /dev/null
```
Example 3
```plain
+ llvm-cov-22 report ut_Date_Time -object ut_Unit_Conversion -object ut_Isolette -object ut_Room_Air -instr-profile=Overall.profdata -dump -num-threads=4
LLVM ERROR: IO failure on output stream: Bad address
```
Example 4
```plain
corrupted size vs. prev_size
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0 libLLVM.so.20.1 0x000078adcac6343f llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 63
1 libLLVM.so.20.1 0x000078adcac61149 llvm::sys::RunSignalHandlers() + 89
2 libLLVM.so.20.1 0x000078adcac63b50
3 libc.so.6 0x000078adc9602330
4 libc.so.6 0x000078adc965bb2c pthread_kill + 284
5 libc.so.6 0x000078adc960227e gsignal + 30
6 libc.so.6 0x000078adc95e58ff abort + 223
7 libc.so.6 0x000078adc95e67b6
8 libc.so.6 0x000078adc9665ff5
9 libc.so.6 0x000078adc9666b96
10 libc.so.6 0x000078adc96680e3
11 libc.so.6 0x000078adc9668529
12 libc.so.6 0x000078adc966adce __libc_free + 126
13 libLLVM.so.20.1 0x000078adcac4256b llvm::raw_ostream::SetBuffered() + 91
14 libLLVM.so.20.1 0x000078adcac432b7 llvm::raw_ostream::write(char const*, unsigned long) + 167
15 llvm-cov-20 0x000058550904c6cf
16 llvm-cov-20 0x000058550904f50a
17 llvm-cov-20 0x000058550903a10f
18 llvm-cov-20 0x000058550903a060
19 libc.so.6 0x000078adc965eed3
20 llvm-cov-20 0x000058550903a54e
21 llvm-cov-20 0x000058550903a5fe
22 libLLVM.so.20.1 0x000078adcac044df llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) + 655
23 libLLVM.so.20.1 0x000078adcac05aa8
24 libc.so.6 0x000078adc9659aa4
25 libc.so.6 0x000078adc96e6a64 __clone + 68
./test_coverage.sh: line 164: 10466 Aborted (core dumped) llvm-cov-20 report "${overall_objects[@]}" -instr-profile=Overall.profdata --show-mcdc-summary "${common_opt[@]}" -dump > "$output_txt"
```
## Workaround
Passing `-num-threads=1` avoids the crash in my testing.
[llvm-cov-reproducer.zip](https://github.com/user-attachments/files/29794220/llvm-cov-reproducer.zip)
Contributor guide
Research direction
Start by running reproducer/crash.sh and the minimized llvm-cov report command with multiple threads. Use the reported CoverageReport::prepareSingleFileReport stack frame as the entry point and compare behavior with -num-threads=1. Done means the reproducer completes reliably in parallel without crashes, heap corruption, or invalid output.
Written by the indexing model from the issue text.
Assessment
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100