Client thread parked forever with queued ThreadlessExecutor tasks
- Vorherrschende Sprache
- Java
- Sterne
- 12.1k
- Forks
- 4k
- Ø Merge
- 2 T. 17 Std.
- Gemergte PRs (30 T.)
- 37
Beschreibung
### What version of gRPC-Java are you using?
1.76.0
### What is your environment?
- **OS**: Linux (aarch64/ARM64)
- **JDK**: openjdk version 21.0.4
### What did you expect to see?
Client thread should not block indefinitely when NettyStream receives both MessageReceive and StreamClose events. blockingUnaryCall should return once the stream is closed and pending callbacks are processed
### What did you see instead?
blockingUnaryCall hangs indefinitely. The client thread remains parked in LockSupport.park() inside ThreadlessExecutor.waitAndDrain() and never resumes execution. This occurs despite pending tasks being present in the ThreadlessExecutor queue, including a SerializingExecutor runnable, and additional queued work in SerializingExecutor.runQueue such as MessagesAvailable and StreamClosed. As a result, these callbacks are never processed and the call remains blocked forever.
**Heap dump analysis:**
| Object | State |
|-----------|-------|
| `Client Thread` | `LockSupport.park()` in `waitAndDrain()` |
| `ThreadlessExecutor.waiter` | The parked thread (correctly set) |
| `ThreadlessExecutor` queue | `[null] (head) → [SerializingExecutor]` (item present but not polled) |
| `SerializingExecutor.runState` | `-1` (RUNNING) |
| `SerializingExecutor.runQueue` | `[null] (head) → [MessagesAvailable] → [StreamClosed]`
The `SerializingExecutor` was present in the `ThreadlessExecutor` queue waiting to be polled, but the parked thread never retrieved it despite `unpark()` having been called. This caused the `StreamClosed` callback to never execute, leaving the blocking call stuck forever.
### Steps to reproduce the bug
Not able to reproduce this issue on demand. It appears to be a rare race condition that has only been observed a few times in production. Also after first occurrence, we upgraded gRPC from 1.64.2 to 1.76.0, but it was still observed afterward.
Beitragsleitfaden
Rechercherichtung
Beginne damit, blockingUnaryCall durch NettyStream zu verfolgen, wenn MessageReceive und StreamClose eintreffen, und untersuche anschließend die Behandlung der Warteschlange in ThreadlessExecutor.waitAndDrain und SerializingExecutor. Verwende den heap-dump-Zustand als Fehlerszenario; abgeschlossen bedeutet, dass StreamClosed und ausstehende Callbacks verarbeitet werden, sodass der blockierende Aufruf zurückkehrt, anstatt geparkt zu bleiben.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- grpc, java
- Bereich
- backend-api-design, networking
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 42/100