eclipse-vertx / eclipse-vertx/vertx-grpc

Deadlock between `WriteStreamAdapter` monitor and the HTTP/2 connection monitor when writing from a non-connection thread

Closed
#363 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
53
Forks
36
Avg merge
7h 37m
Merged PRs (30d)
64

Description

### Version

4.5.32 (via quarkus 3.38.3)

### Context

Two event loops stopped simultaneously and never recovered (~90 s, until the JVM was killed by its
Kubernetes liveness probe). `BlockedThreadChecker` kept reporting, so the JVM itself was not stalled —
GC overhead was 0.24% with zero major collections, and there were only 6 in-flight requests.

The deadlock in `WriteStreamAdapter`:

Let **A** = the `WriteStreamAdapter` instance monitor, **B** = the HTTP/2 connection monitor
(every method on `Http2ServerResponse` is `synchronized (conn)`).

**Order A → B** — `WriteStreamAdapter` takes its own monitor, then reaches into the stream:

```java
public final void write(T msg) {
stream.writeMessage(encoder.encode(msg, wireFormat));
synchronized (this) { // takes A
ready = !stream.writeQueueFull(); // -> Http2ServerResponse.writeQueueFull() -> synchronized (conn) == takes B
}
}

private void checkReady() {
synchronized (this) { // takes A
if (ready || stream.writeQueueFull()) { // -> synchronized (conn) == takes B
return;
}
ready = true;
}
handleReady();
}
```

Thread 1 — holds **B** (inside `Http2ServerResponse.write`), blocked taking **A** at `checkReady`:

```
io.vertx.grpc.common.impl.WriteStreamAdapter.checkReady(WriteStreamAdapter.java:54) <- synchronized (this)
io.vertx.grpc.common.impl.WriteStreamAdapter.lambda$init$0(WriteStreamAdapter.java:37)
io.vertx.core.http.impl.Http2ServerResponse.handlerWritabilityChanged(Http2ServerResponse.java:575)
io.vertx.core.http.impl.Http2ServerStream.handleWritabilityChanged(Http2ServerStream.java:171)
io.vertx.core.http.impl.VertxHttp2Stream.onWritabilityChanged(VertxHttp2Stream.java:127)
io.vertx.core.http.impl.Http2ConnectionBase.onStreamWritabilityChanged(Http2ConnectionBase.java:144)
io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$ListenerWritabilityMonitor.notifyWritabilityChanged(...)
io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.forEachActiveStream(DefaultHttp2Connection.java:1014)
io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$ListenerWritabilityMonitor.checkAllWritabilityChanged(...)
io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:383)
io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:437)
...
io.vertx.core.http.impl.VertxHttp2Stream.writeData(VertxHttp2Stream.java:236)
io.vertx.core.http.impl.Http2ServerResponse.write(Http2ServerResponse.java:477) <- holds synchronized (conn)
io.vertx.grpc.server.impl.GrpcServerResponseImpl.writeMessage(GrpcServerResponseImpl.java:297)
io.vertx.grpc.common.impl.WriteStreamAdapter.write(WriteStreamAdapter.java:47)
io.vertx.grpc.server.impl.GrpcServiceBridgeImpl$ServerCallImpl.sendMessage(GrpcServiceBridgeImpl.java:187)
```

Thread 2 — holds **A**, blocked taking **B** inside `writeQueueFull()`:

```
io.vertx.grpc.common.impl.WriteStreamAdapter.write(WriteStreamAdapter.java:49) <- inside synchronized (this),
calling stream.writeQueueFull()
```

This race has occured a few times in production for us, but after reducing the http2 message response size, it seems to have dissapeared, however, it would be nice to not be that carefull about the size and what threads actually sends it back to the client.

### Steps to reproduce

1. Two or more concurrent gRPC calls multiplexed on one HTTP/2 connection.
2. A response large enough to exceed the channel's high-water mark, so that completing a write flips
writability back and triggers `forEachActiveStream`.
3. `sendMessage` invoked from a thread that is not the connection's event loop.

### Do you have a reproducer?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with io.vertx.grpc.common.impl.WriteStreamAdapter, especially write(), checkReady(), and the init callback, then trace the referenced Http2ServerResponse methods and the two stack traces. Reproduce the concurrent HTTP/2 scenario with a large response and a non-connection-thread sendMessage call; done means the event loops no longer deadlock under those conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
backend-api-design, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.