SIGPROF backtrace deadlocks during exception unwinding
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
We encountered a deadlock in a FoundationDB 7.4.6 release build. The affected fdbserver remained alive but stopped processing requests, and cluster recovery stalled waiting for its TLog.
Environment:
- Linux x86_64, Ubuntu, kernel 5.15.0-161-generic
- FoundationDB 7.4.6
We inspected the live process using GDB and found:
1. Connection::read() encountered a socket EOF and threw connection_failed() (error code 1026). We verified both the exception object and the underlying Boost.Asio error (misc_category, value 2).
2. During exception unwinding, the main thread entered _Unwind_Find_FDE() in libgcc_s.so.1.
3. The thread was interrupted by the run-loop profiler’s SIGPROF handler, which called libc backtrace().
4. backtrace() re-entered _Unwind_Find_FDE() and blocked on a mutex already owned by the same main
thread.
Relevant stack frames, with unresolved FDB frames omitted:
```
pthread_mutex_lock
_Unwind_Find_FDE
...
_Unwind_Backtrace
__GI___backtrace
...
_Unwind_Find_FDE
...
_Unwind_RaiseException
```
The mutex state confirmed a self-deadlock:
Main thread TID: 1295175
```
__lock = 2
__owner = 1295175
__kind = 0
```
The process also had an fdb-loopprofile thread. Inspection of the running binary confirmed calls to
pthread_kill(mainThread, SIGPROF) and a configured run-loop profiling interval of 0.125 seconds.
Related reports and changes:
- https://github.com/apple/foundationdb/pull/9953
- https://github.com/apple/foundationdb/pull/1792
- https://github.com/apple/foundationdb/pull/7374
- https://forums.foundationdb.org/t/deadlock-caused-by-profiler-thread-in-storage-server/3921
PR #9953 describes the same deadlock mechanism, but its fix specifically addresses sanitizer builds.
Our build does not use sanitizers.
Contributor guide
Research direction
Start with the run-loop profiler path that calls pthread_kill(mainThread, SIGPROF), and compare its non-sanitizer behavior with PR #9953 and the related profiler changes. Use the supplied GDB stack and FoundationDB 7.4.6 environment to define a reproducible case; done means the profiling path no longer leaves exception unwinding deadlocked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, linux
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100