grpc / grpc/grpc-java

Client thread parked forever with queued ThreadlessExecutor tasks

Open
#12,648 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
12.1k
Forks
4k
Avg merge
2d 17h
Merged PRs (30d)
37

Description

### What version of gRPC-Java are you using?
1.76.0

### What is your environment?
- **OS**: Linux (aarch64/ARM64)
- **JDK**: openjdk version 21.0.4

### What did you expect to see?

Client thread should not block indefinitely when NettyStream receives both MessageReceive and StreamClose events. blockingUnaryCall should return once the stream is closed and pending callbacks are processed

### What did you see instead?

blockingUnaryCall hangs indefinitely. The client thread remains parked in LockSupport.park() inside ThreadlessExecutor.waitAndDrain() and never resumes execution. This occurs despite pending tasks being present in the ThreadlessExecutor queue, including a SerializingExecutor runnable, and additional queued work in SerializingExecutor.runQueue such as MessagesAvailable and StreamClosed. As a result, these callbacks are never processed and the call remains blocked forever.

**Heap dump analysis:**

| Object | State |
|-----------|-------|
| `Client Thread` | `LockSupport.park()` in `waitAndDrain()` |
| `ThreadlessExecutor.waiter` | The parked thread (correctly set) |
| `ThreadlessExecutor` queue | `[null] (head) → [SerializingExecutor]` (item present but not polled) |
| `SerializingExecutor.runState` | `-1` (RUNNING) |
| `SerializingExecutor.runQueue` | `[null] (head) → [MessagesAvailable] → [StreamClosed]`

The `SerializingExecutor` was present in the `ThreadlessExecutor` queue waiting to be polled, but the parked thread never retrieved it despite `unpark()` having been called. This caused the `StreamClosed` callback to never execute, leaving the blocking call stuck forever.

### Steps to reproduce the bug
Not able to reproduce this issue on demand. It appears to be a rare race condition that has only been observed a few times in production. Also after first occurrence, we upgraded gRPC from 1.64.2 to 1.76.0, but it was still observed afterward.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.