grpc / grpc/grpc-java

Client thread parked forever with queued ThreadlessExecutor tasks

Aperta
#12,648 9 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Java
Stelle
12.1k
Fork
4k
Merge medio
2g 17h
PR unite (30g)
37

Descrizione

### What version of gRPC-Java are you using?
1.76.0

### What is your environment?
- **OS**: Linux (aarch64/ARM64)
- **JDK**: openjdk version 21.0.4

### What did you expect to see?

Client thread should not block indefinitely when NettyStream receives both MessageReceive and StreamClose events. blockingUnaryCall should return once the stream is closed and pending callbacks are processed

### What did you see instead?

blockingUnaryCall hangs indefinitely. The client thread remains parked in LockSupport.park() inside ThreadlessExecutor.waitAndDrain() and never resumes execution. This occurs despite pending tasks being present in the ThreadlessExecutor queue, including a SerializingExecutor runnable, and additional queued work in SerializingExecutor.runQueue such as MessagesAvailable and StreamClosed. As a result, these callbacks are never processed and the call remains blocked forever.

**Heap dump analysis:**

| Object | State |
|-----------|-------|
| `Client Thread` | `LockSupport.park()` in `waitAndDrain()` |
| `ThreadlessExecutor.waiter` | The parked thread (correctly set) |
| `ThreadlessExecutor` queue | `[null] (head) → [SerializingExecutor]` (item present but not polled) |
| `SerializingExecutor.runState` | `-1` (RUNNING) |
| `SerializingExecutor.runQueue` | `[null] (head) → [MessagesAvailable] → [StreamClosed]`

The `SerializingExecutor` was present in the `ThreadlessExecutor` queue waiting to be polled, but the parked thread never retrieved it despite `unpark()` having been called. This caused the `StreamClosed` callback to never execute, leaving the blocking call stuck forever.

### Steps to reproduce the bug
Not able to reproduce this issue on demand. It appears to be a rare race condition that has only been observed a few times in production. Also after first occurrence, we upgraded gRPC from 1.64.2 to 1.76.0, but it was still observed afterward.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia tracciando blockingUnaryCall attraverso NettyStream quando arrivano MessageReceive e StreamClose, quindi esamina la gestione della coda di ThreadlessExecutor.waitAndDrain e SerializingExecutor. Usa lo stato dell’heap-dump come scenario di errore; il lavoro è completato quando StreamClosed e i callback in attesa vengono elaborati, così che la chiamata bloccante ritorni invece di rimanere parcheggiata.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
grpc, java
Ambito
backend-api-design, networking
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
42/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.