[rpc] Missing client-side request deadlines can stall Flink writes until connection idle timeout
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Problem
Java client RPCs have no independent timeout while pending or in flight. If a request receives neither a response nor a connection failure, its future can remain incomplete until the connection closes, delaying retries and blocking Flink sink flush/checkpoint completion during a rolling upgrade.
For writes, `client.request-timeout` (default 30 seconds) is sent to the server for its acknowledgement wait; it does not enforce a client-side response deadline. Recovery may therefore wait for the default 10-minute connection idle timeout, or longer if other traffic keeps the connection active.
### Example reproduction
A transport-level reproduction, without requiring a rolling upgrade:
1. Start a localhost `NettyClient` and `NettyServer`. Set `client.request-timeout=100 ms`, client `netty.connection.max-idle-time=2 s`, and server idle timeout to 30 seconds.
2. Allow version negotiation to finish so the connection becomes ready.
3. Send an `API_VERSIONS` request whose test-server handler returns a deliberately uncompleted future. Keep the connection open.
4. After 500 ms, the client future is still incomplete and the request remains in `inflightRequests`, despite the configured 100 ms request timeout.
5. At approximately 2 seconds, the client closes the idle connection. Only then does the request fail with `NetworkException` and leave the in-flight map.
Expected: the RPC should expire through a client-side deadline and release its state independently of connection idleness.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Java NettyClient request path and its inflightRequests handling, using the API_VERSIONS reproduction described in the issue. Verify the configured 100 ms client request timeout against the 2-second connection idle timeout. Done means an unanswered RPC fails at its client-side deadline and is removed from in-flight state without waiting for connection closure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100