apple / apple/foundationdb

SIGPROF backtrace deadlocks during exception unwinding

Open
#14,022 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

We encountered a deadlock in a FoundationDB 7.4.6 release build. The affected fdbserver remained alive but stopped processing requests, and cluster recovery stalled waiting for its TLog.

Environment:

- Linux x86_64, Ubuntu, kernel 5.15.0-161-generic
- FoundationDB 7.4.6

We inspected the live process using GDB and found:

1. Connection::read() encountered a socket EOF and threw connection_failed() (error code 1026). We verified both the exception object and the underlying Boost.Asio error (misc_category, value 2).

2. During exception unwinding, the main thread entered _Unwind_Find_FDE() in libgcc_s.so.1.
3. The thread was interrupted by the run-loop profiler’s SIGPROF handler, which called libc backtrace().

4. backtrace() re-entered _Unwind_Find_FDE() and blocked on a mutex already owned by the same main
thread.

Relevant stack frames, with unresolved FDB frames omitted:

```
pthread_mutex_lock
_Unwind_Find_FDE
...
_Unwind_Backtrace
__GI___backtrace


...
_Unwind_Find_FDE
...
_Unwind_RaiseException

```

The mutex state confirmed a self-deadlock:

Main thread TID: 1295175

```
__lock = 2
__owner = 1295175
__kind = 0
```

The process also had an fdb-loopprofile thread. Inspection of the running binary confirmed calls to
pthread_kill(mainThread, SIGPROF) and a configured run-loop profiling interval of 0.125 seconds.

Related reports and changes:

- https://github.com/apple/foundationdb/pull/9953
- https://github.com/apple/foundationdb/pull/1792
- https://github.com/apple/foundationdb/pull/7374
- https://forums.foundationdb.org/t/deadlock-caused-by-profiler-thread-in-storage-server/3921

PR #9953 describes the same deadlock mechanism, but its fix specifically addresses sanitizer builds.
Our build does not use sanitizers.

Contributor guide

Open the contributing guide

Research direction

Start with the run-loop profiler path that calls pthread_kill(mainThread, SIGPROF), and compare its non-sanitizer behavior with PR #9953 and the related profiler changes. Use the supplied GDB stack and FoundationDB 7.4.6 environment to define a reproducible case; done means the profiling path no longer leaves exception unwinding deadlocked.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
observability-sre
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.