eclipse-vertx / eclipse-vertx/vertx-grpc
Deadlock between `WriteStreamAdapter` monitor and the HTTP/2 connection monitor when writing from a non-connection thread
- Dominant language
- Java
- Stars
- 53
- Forks
- 36
- Avg merge
- 7h 37m
- Merged PRs (30d)
- 64
Description
### Version
4.5.32 (via quarkus 3.38.3)
### Context
Two event loops stopped simultaneously and never recovered (~90 s, until the JVM was killed by its
Kubernetes liveness probe). `BlockedThreadChecker` kept reporting, so the JVM itself was not stalled —
GC overhead was 0.24% with zero major collections, and there were only 6 in-flight requests.
The deadlock in `WriteStreamAdapter`:
Let **A** = the `WriteStreamAdapter` instance monitor, **B** = the HTTP/2 connection monitor
(every method on `Http2ServerResponse` is `synchronized (conn)`).
**Order A → B** — `WriteStreamAdapter` takes its own monitor, then reaches into the stream:
```java
public final void write(T msg) {
stream.writeMessage(encoder.encode(msg, wireFormat));
synchronized (this) { // takes A
ready = !stream.writeQueueFull(); // -> Http2ServerResponse.writeQueueFull() -> synchronized (conn) == takes B
}
}
private void checkReady() {
synchronized (this) { // takes A
if (ready || stream.writeQueueFull()) { // -> synchronized (conn) == takes B
return;
}
ready = true;
}
handleReady();
}
```
Thread 1 — holds **B** (inside `Http2ServerResponse.write`), blocked taking **A** at `checkReady`:
```
io.vertx.grpc.common.impl.WriteStreamAdapter.checkReady(WriteStreamAdapter.java:54) <- synchronized (this)
io.vertx.grpc.common.impl.WriteStreamAdapter.lambda$init$0(WriteStreamAdapter.java:37)
io.vertx.core.http.impl.Http2ServerResponse.handlerWritabilityChanged(Http2ServerResponse.java:575)
io.vertx.core.http.impl.Http2ServerStream.handleWritabilityChanged(Http2ServerStream.java:171)
io.vertx.core.http.impl.VertxHttp2Stream.onWritabilityChanged(VertxHttp2Stream.java:127)
io.vertx.core.http.impl.Http2ConnectionBase.onStreamWritabilityChanged(Http2ConnectionBase.java:144)
io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$ListenerWritabilityMonitor.notifyWritabilityChanged(...)
io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.forEachActiveStream(DefaultHttp2Connection.java:1014)
io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$ListenerWritabilityMonitor.checkAllWritabilityChanged(...)
io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:383)
io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:437)
...
io.vertx.core.http.impl.VertxHttp2Stream.writeData(VertxHttp2Stream.java:236)
io.vertx.core.http.impl.Http2ServerResponse.write(Http2ServerResponse.java:477) <- holds synchronized (conn)
io.vertx.grpc.server.impl.GrpcServerResponseImpl.writeMessage(GrpcServerResponseImpl.java:297)
io.vertx.grpc.common.impl.WriteStreamAdapter.write(WriteStreamAdapter.java:47)
io.vertx.grpc.server.impl.GrpcServiceBridgeImpl$ServerCallImpl.sendMessage(GrpcServiceBridgeImpl.java:187)
```
Thread 2 — holds **A**, blocked taking **B** inside `writeQueueFull()`:
```
io.vertx.grpc.common.impl.WriteStreamAdapter.write(WriteStreamAdapter.java:49) <- inside synchronized (this),
calling stream.writeQueueFull()
```
This race has occured a few times in production for us, but after reducing the http2 message response size, it seems to have dissapeared, however, it would be nice to not be that carefull about the size and what threads actually sends it back to the client.
### Steps to reproduce
1. Two or more concurrent gRPC calls multiplexed on one HTTP/2 connection.
2. A response large enough to exceed the channel's high-water mark, so that completing a write flips
writability back and triggers `forEachActiveStream`.
3. `sendMessage` invoked from a thread that is not the connection's event loop.
### Do you have a reproducer?
_No response_
Contributor guide
Research direction
Start with io.vertx.grpc.common.impl.WriteStreamAdapter, especially write(), checkReady(), and the init callback, then trace the referenced Http2ServerResponse methods and the two stack traces. Reproduce the concurrent HTTP/2 scenario with a large response and a non-connection-thread sendMessage call; done means the event loops no longer deadlock under those conditions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java
- Domain
- backend-api-design, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100