llvm / llvm/llvm-project

[llvm-cov] `llvm-cov report` intermittently crashes with `-num-threads > 1`

Open
#208,182 0 comments 0 reactions 0 assignees View on GitHub
crash tools:llvm-cov
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

`llvm-cov-20 report` intermittently crashes when generating a summary from multiple instrumented executables. The crash only appears when more than one worker thread is used. `-num-threads=1` seems to be a reliable workaround.

## Environment

I have currently tested this behavior on Ubuntu 24.04 with LLVM 20.1.8 and 22.1.8.

## Reproducer

The attached `reproducer` directory contains:

- `Overall.profdata`
- four instrumented test executables with debug information
- `crash.sh`, which repeatedly runs `llvm-cov-20 report`

Run:

```sh
cd reproducer
./crash.sh
```

The minimized command is:

```sh
llvm-cov-22 report \
ut_Date_Time \
-object ut_Unit_Conversion \
-object ut_Isolette \
-object ut_Room_Air \
-instr-profile=Overall.profdata \
-dump \
-num-threads=8 \
> /dev/null
```

The crash is nondeterministic. On my machine it usually happens within a few iterations with `-num-threads=$(nproc)`.

## Expected behavior

`llvm-cov report` should either complete successfully or report a normal diagnostic. It should not crash or corrupt heap state.

## Actual behavior

The failure is nondeterministic and the exact symptom varies. I have observed heap corruption diagnostics, segmentation faults, aborts, and in one run the LLVM crash-report message was printed but the process did not exit at all.

A few representative crashes are shown below.

Example 1

```plain
+ llvm-cov-22 report ut_Date_Time -object ut_Unit_Conversion -object ut_Isolette -object ut_Room_Air -instr-profile=Overall.profdata -dump -num-threads=4
double free or corruption (!prev)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
#0 0x00007588094dd17d llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xcdd17d)
#1 0x00007588094da120 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
#2 0x0000758808045330 (/lib/x86_64-linux-gnu/libc.so.6+0x45330)
#3 0x000075880809eb2c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
#4 0x000075880809eb2c __pthread_kill_internal ./nptl/pthread_kill.c:78:10
#5 0x000075880809eb2c pthread_kill ./nptl/pthread_kill.c:89:10
#6 0x000075880804527e raise ./signal/../sysdeps/posix/raise.c:27:6
#7 0x00007588080288ff abort ./stdlib/abort.c:81:7
#8 0x00007588080297b6 _IO_acquire_lock_fct ./libio/libioP.h:979:6
#9 0x00007588080297b6 _IO_peekc_locked ./libio/peekc.c:37:3
#10 0x00007588080a8ff5 ./malloc/malloc.c:5775:3
#11 0x00007588080ab12c _int_free_merge_chunk ./malloc/malloc.c:4679:5
#12 0x00007588080ab529 _int_free ./malloc/malloc.c:4648:5
#13 0x00007588080addce cfree ./malloc/malloc.c:3401:3
#14 0x00007588094adae7 llvm::raw_ostream::SetBuffered() (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xcadae7)
#15 0x00007588094ae537 llvm::raw_ostream::write(char const*, unsigned long) (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xcae537)
#16 0x000058ab3f596306 llvm::CoverageReport::prepareSingleFileReport(llvm::StringRef, llvm::coverage::CoverageMapping const*, llvm::CoverageViewOptions const&, unsigned int, llvm::FileCoverageSummary*, llvm::CoverageFilter const*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x48306)
#17 0x000058ab3f594f1d void llvm::detail::UniqueFunctionBase::CallImpl>(void*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x46f1d)
#18 0x000058ab3f5652e7 std::_Function_handler (), std::__future_base::_Task_setter, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker>>, void>>::_M_invoke(std::_Any_data const&) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x172e7)
#19 0x000058ab3f567249 std::__future_base::_State_baseV2::_M_do_set(std::function ()>*, bool*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x19249)
#20 0x00007588080a1ed3 __pthread_once_slow ./nptl/pthread_once.c:118:7
#21 0x000058ab3f56719c std::__future_base::_Deferred_state>>, void>::_M_complete_async() (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x1919c)
#22 0x000058ab3f5664ec void llvm::detail::UniqueFunctionBase::CallImpl llvm::ThreadPoolInterface::asyncImpl(llvm::unique_function, llvm::ThreadPoolTaskGroup*)::'lambda'()>(void*) (/home/luca/micromamba/envs/clang-22/bin/llvm-cov-22+0x184ec)
#23 0x000075880944d639 llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/home/luca/micromamba/envs/clang-22/bin/../lib/libLLVM.so.22.1+0xc4d639)
#24 0x000075880944e2e7 void* llvm::thread::ThreadProxy>(void*) ThreadPool.cpp:0:0
#25 0x000075880809caa4 start_thread ./nptl/pthread_create.c:447:8
#26 0x0000758808129c6c clone3 ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:80:0
./crash.sh: line 20: 214489 Aborted llvm-cov-22 report "${overall_objects[@]}" -instr-profile=Overall.profdata -dump "-num-threads=4" > /dev/null
```

Example 2

```plain
double free or corruption (out)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
malloc(): unaligned tcache chunk detected
./crash.sh: line 20: 217872 Aborted llvm-cov-22 report "${overall_objects[@]}" -instr-profile=Overall.profdata -dump "-num-threads=4" > /dev/null
```

Example 3

```plain
+ llvm-cov-22 report ut_Date_Time -object ut_Unit_Conversion -object ut_Isolette -object ut_Room_Air -instr-profile=Overall.profdata -dump -num-threads=4
LLVM ERROR: IO failure on output stream: Bad address
```

Example 4

```plain
corrupted size vs. prev_size
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0 libLLVM.so.20.1 0x000078adcac6343f llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 63
1 libLLVM.so.20.1 0x000078adcac61149 llvm::sys::RunSignalHandlers() + 89
2 libLLVM.so.20.1 0x000078adcac63b50
3 libc.so.6 0x000078adc9602330
4 libc.so.6 0x000078adc965bb2c pthread_kill + 284
5 libc.so.6 0x000078adc960227e gsignal + 30
6 libc.so.6 0x000078adc95e58ff abort + 223
7 libc.so.6 0x000078adc95e67b6
8 libc.so.6 0x000078adc9665ff5
9 libc.so.6 0x000078adc9666b96
10 libc.so.6 0x000078adc96680e3
11 libc.so.6 0x000078adc9668529
12 libc.so.6 0x000078adc966adce __libc_free + 126
13 libLLVM.so.20.1 0x000078adcac4256b llvm::raw_ostream::SetBuffered() + 91
14 libLLVM.so.20.1 0x000078adcac432b7 llvm::raw_ostream::write(char const*, unsigned long) + 167
15 llvm-cov-20 0x000058550904c6cf
16 llvm-cov-20 0x000058550904f50a
17 llvm-cov-20 0x000058550903a10f
18 llvm-cov-20 0x000058550903a060
19 libc.so.6 0x000078adc965eed3
20 llvm-cov-20 0x000058550903a54e
21 llvm-cov-20 0x000058550903a5fe
22 libLLVM.so.20.1 0x000078adcac044df llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) + 655
23 libLLVM.so.20.1 0x000078adcac05aa8
24 libc.so.6 0x000078adc9659aa4
25 libc.so.6 0x000078adc96e6a64 __clone + 68
./test_coverage.sh: line 164: 10466 Aborted (core dumped) llvm-cov-20 report "${overall_objects[@]}" -instr-profile=Overall.profdata --show-mcdc-summary "${common_opt[@]}" -dump > "$output_txt"
```

## Workaround

Passing `-num-threads=1` avoids the crash in my testing.

[llvm-cov-reproducer.zip](https://github.com/user-attachments/files/29794220/llvm-cov-reproducer.zip)

Contributor guide

Open the contributing guide

Research direction

Start by running reproducer/crash.sh and the minimized llvm-cov report command with multiple threads. Use the reported CoverageReport::prepareSingleFileReport stack frame as the entry point and compare behavior with -num-threads=1. Done means the reproducer completes reliably in parallel without crashes, heap corruption, or invalid output.

Written by the indexing model from the issue text.

Assessment

Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.