absl::Mutex deadlock-detector abort in SecurityHandshaker::DoHandshake() under connection churn (grpc 1.74.0, EventEngine posix engine)
- Dominant language
- C++
- Stars
- 45.3k
- Forks
- 11.4k
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 47
Description
### What version of gRPC and what language are you using?
grpc 1.74.0 (upstream tarball build), C++ (crashing server). Reproduction driver
below is Python (`grpcio`) acting as the client, but the crash itself is inside
the C++ core library used by our C++ server.
### What operating system (Linux, Windows,...) and version?
Linux, RHEL 10.2, kernel `6.12.0-211.32.1.el10_2`, x86_64.
### What runtime / compiler are you using (e.g. python version or version of gcc)
Server: C++, built against grpc 1.74.0 and Abseil `lts_20240722`, with `NDEBUG`
undefined (so `absl::Mutex`'s deadlock detector is compiled in and active).
Client (reproduction driver only): Python 3 + `grpcio`.
### What did you do?
Server: a standard gRPC C++ server (`grpc::Server` + `SyncRequestThreadManager`,
the synchronous API) exposing one simple unary RPC, listening on a local TCP port.
Client: ~16 concurrent copies of the Python script below, each looping for a fixed
duration. Each iteration opens a **brand-new plaintext/insecure channel**
(`grpc.insecure_channel`, no TLS credentials), issues one unary call, then lets the
`with` block close the channel — i.e. connect, one RPC, disconnect, repeat, as fast
as possible. (Note: even though the channel is "insecure"/plaintext, it still goes
through `SecurityHandshaker` — grpc routes every connection, TLS or not, through
the same security-handshake abstraction; a plaintext channel just uses a no-op
local security connector instead of a real TLS one, so `SecurityHandshaker`'s
code paths still run on every connection churn cycle.)
```python
import sys, time, grpc
import your_service_pb2 as pb2
import your_service_pb2_grpc as pb2_grpc
def main():
worker_id = int(sys.argv[1])
duration = float(sys.argv[2])
deadline = time.time() + duration
count = 0
errors = 0
req_id = 0
while time.time() < deadline:
req_id += 1
try:
with grpc.insecure_channel("127.0.0.1:") as channel:
stub = pb2_grpc.YourServiceStub(channel)
stub.YourUnaryMethod(pb2.YourRequest(request_id=req_id), timeout=5)
count += 1
except Exception:
errors += 1
print(f"worker {worker_id}: {count} ok, {errors} errors")
if __name__ == "__main__":
main()
```
Run 16 of these concurrently against the server, e.g.:
```bash
for i in $(seq 1 16); do
python3 load_gen.py "$i" 300 > "worker_${i}.log" 2>&1 &
done
wait
```
(We reproduced this against one of our own internal unary RPCs — the exact
service/method is not load-bearing; any simple unary call reproduces it. What
matters is the pattern: many concurrent short-lived plaintext channels, each doing
exactly one RPC before disconnecting.) This reliably crashes the server within
seconds to tens of seconds. At idle, with the load stopped, the server is stable —
this is specifically a connection-churn trigger.
We also tried, neither of which changed the outcome:
- `GRPC_POLL_STRATEGY=epoll1` (vs. default `poll`) — no effect.
- `GRPC_EXPERIMENTS=-event_engine_client,-event_engine_listener,-event_engine_for_all_other_endpoints,-event_engine_dns`
— no effect.
The only thing that stopped the crash was calling
`absl::SetMutexDeadlockDetectionMode(absl::OnDeadlockCycle::kReport)` in our own
process — this converts the abort into a log-and-continue, matching the mitigation
mentioned in #40390, but it's a workaround, not a fix for whatever the detector is
flagging.
### What did you expect to see?
The server continuing to serve RPCs normally under this connection-churn pattern
(many short-lived plaintext connections), without the process aborting.
### What did you see instead?
The process aborts via `absl::Mutex`'s `DebugOnlyDeadlockCheck()` — "cycle in the
historical lock ordering graph has been observed," the same abort mechanism
reported in #40390 / #40378. Functions involved in the flagged lock-order cycle:
- `grpc_core::(anonymous namespace)::SecurityHandshaker::DoHandshake()`
- `grpc_event_engine::experimental::PosixEndpointImpl::MaybeShutdown()`
- `grpc_event_engine::experimental::PosixEndpointImpl::MaybePostReclaimer()`
- `grpc_core::GrpcMemoryAllocatorImpl::Shutdown()`
- `grpc_event_engine::experimental::(anonymous namespace)::EventEngineEndpointWrapper::TriggerShutdown()`
We confirmed this is not the low-level poller: switching `GRPC_POLL_STRATEGY`
between `poll`/`epoll1`, and toggling the `event_engine_client` /
`event_engine_listener` / `event_engine_for_all_other_endpoints` /
`event_engine_dns` experiments, has no effect — identical crash signature in every
combination. The bug appears to live in the shared handshake/endpoint-teardown
machinery above the poller, not in the poller itself.
### Anything else we should know about your project / environment?
This is the same general abort mechanism as #40390 / #40378 / the now-closed
#43184 — but all three of those are on the `TCPConnectHandshaker` path. This
combination of functions, on the `SecurityHandshaker` path, doesn't appear to be
reported elsewhere:
- [#40390](https://github.com/grpc/grpc/issues/40390) — same abort mechanism, but
`TCPConnectHandshaker::DoHandshake()` + `PosixEndpointImpl::MaybeShutdown()`, not
`SecurityHandshaker`.
- [#40378](https://github.com/grpc/grpc/issues/40378) — same as above,
`TCPConnectHandshaker`-based.
- [#37982](https://github.com/grpc/grpc/issues/37982) — `PosixEndpointImpl`, but a
different function (`ZerocopyDisableAndWaitForRemaining()`); no mention of
`SecurityHandshaker` or `GrpcMemoryAllocatorImpl`.
- [#43184](https://github.com/grpc/grpc/pull/43184) — proposed fix for
`TCPConnectHandshaker`, closed unmerged; author agreed with maintainer
`markdroth` that the flagged cycle there is a historical-ordering artifact, not
a live deadlock, in that specific case.
Is this a known/tracked pattern for `SecurityHandshaker` specifically, or worth a
closer look given it's a distinct combination of functions from the existing
reports above?
Contributor guide
Research direction
Reproduce the crash with the listed 16-client connection-churn driver against a synchronous C++ server, then trace SecurityHandshaker::DoHandshake() alongside PosixEndpointImpl::MaybeShutdown(), MaybePostReclaimer(), GrpcMemoryAllocatorImpl::Shutdown(), and EventEngineEndpointWrapper::TriggerShutdown(). Done means identifying and addressing the reported lock-order failure so the server survives the churn without aborting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- api, backend, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100