grpc / grpc/grpc-java

Client thread parked forever with queued ThreadlessExecutor tasks

Offen
#12,648 9 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Java
Sterne
12.1k
Forks
4k
Ø Merge
2 T. 17 Std.
Gemergte PRs (30 T.)
37

Beschreibung

### What version of gRPC-Java are you using?
1.76.0

### What is your environment?
- **OS**: Linux (aarch64/ARM64)
- **JDK**: openjdk version 21.0.4

### What did you expect to see?

Client thread should not block indefinitely when NettyStream receives both MessageReceive and StreamClose events. blockingUnaryCall should return once the stream is closed and pending callbacks are processed

### What did you see instead?

blockingUnaryCall hangs indefinitely. The client thread remains parked in LockSupport.park() inside ThreadlessExecutor.waitAndDrain() and never resumes execution. This occurs despite pending tasks being present in the ThreadlessExecutor queue, including a SerializingExecutor runnable, and additional queued work in SerializingExecutor.runQueue such as MessagesAvailable and StreamClosed. As a result, these callbacks are never processed and the call remains blocked forever.

**Heap dump analysis:**

| Object | State |
|-----------|-------|
| `Client Thread` | `LockSupport.park()` in `waitAndDrain()` |
| `ThreadlessExecutor.waiter` | The parked thread (correctly set) |
| `ThreadlessExecutor` queue | `[null] (head) → [SerializingExecutor]` (item present but not polled) |
| `SerializingExecutor.runState` | `-1` (RUNNING) |
| `SerializingExecutor.runQueue` | `[null] (head) → [MessagesAvailable] → [StreamClosed]`

The `SerializingExecutor` was present in the `ThreadlessExecutor` queue waiting to be polled, but the parked thread never retrieved it despite `unpark()` having been called. This caused the `StreamClosed` callback to never execute, leaving the blocking call stuck forever.

### Steps to reproduce the bug
Not able to reproduce this issue on demand. It appears to be a rare race condition that has only been observed a few times in production. Also after first occurrence, we upgraded gRPC from 1.64.2 to 1.76.0, but it was still observed afterward.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne damit, blockingUnaryCall durch NettyStream zu verfolgen, wenn MessageReceive und StreamClose eintreffen, und untersuche anschließend die Behandlung der Warteschlange in ThreadlessExecutor.waitAndDrain und SerializingExecutor. Verwende den heap-dump-Zustand als Fehlerszenario; abgeschlossen bedeutet, dass StreamClosed und ausstehende Callbacks verarbeitet werden, sodass der blockierende Aufruf zurückkehrt, anstatt geparkt zu bleiben.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
grpc, java
Bereich
backend-api-design, networking
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
42/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.