apache / apache/fluss

[rpc] Missing client-side request deadlines can stall Flink writes until connection idle timeout

Open
#4,279 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Problem

Java client RPCs have no independent timeout while pending or in flight. If a request receives neither a response nor a connection failure, its future can remain incomplete until the connection closes, delaying retries and blocking Flink sink flush/checkpoint completion during a rolling upgrade.

For writes, `client.request-timeout` (default 30 seconds) is sent to the server for its acknowledgement wait; it does not enforce a client-side response deadline. Recovery may therefore wait for the default 10-minute connection idle timeout, or longer if other traffic keeps the connection active.

### Example reproduction

A transport-level reproduction, without requiring a rolling upgrade:

1. Start a localhost `NettyClient` and `NettyServer`. Set `client.request-timeout=100 ms`, client `netty.connection.max-idle-time=2 s`, and server idle timeout to 30 seconds.
2. Allow version negotiation to finish so the connection becomes ready.
3. Send an `API_VERSIONS` request whose test-server handler returns a deliberately uncompleted future. Keep the connection open.
4. After 500 ms, the client future is still incomplete and the request remains in `inflightRequests`, despite the configured 100 ms request timeout.
5. At approximately 2 seconds, the client closes the idle connection. Only then does the request fail with `NetworkException` and leave the in-flight map.

Expected: the RPC should expire through a client-side deadline and release its state independently of connection idleness.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Java NettyClient request path and its inflightRequests handling, using the API_VERSIONS reproduction described in the issue. Verify the configured 100 ms client request timeout against the 2-second connection idle timeout. Done means an unanswered RPC fails at its client-side deadline and is removed from in-flight state without waiting for connection closure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.