[Client] Add client.lookup.max-inflight-requests-per-bucket to restrict lookup requests per server
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
0.9.0 (latest release)
### Please describe the bug 🐞
`client.lookup.max-inflight-requests` currently provides only a client-wide RPC concurrency limit.
During leader failover, such as a rolling upgrade, requests sent to an old or temporarily unavailable leader may remain incomplete until the connection fails or the request times out.
If many lookups target the same affected bucket, their RPCs may consume most or all global in-flight permits:
`client.lookup.max-inflight-server` (which is same as `writer.max-inflight-requests-per-bucket`
Once the global permits are exhausted, the lookup sender blocks before sending additional RPCs. Requests for healthy buckets and healthy TabletServers can no longer be dispatched, even though they are unrelated to the failed bucket.
To isolate this failure, the lookup client needs an additional per-bucket limit:
`client.lookup.max-inflight-requests-per-bucket`
Once one bucket reaches this limit, further requests for that bucket should remain queued, while requests for other buckets remain eligible for dispatch.
This is specifically per-bucket isolation. Multiple affected buckets on the same failed TabletServer can still collectively consume global permits; the option is not a per-node circuit breaker.
### Solution
* client.lookup.queue-size : Maximum number of admitted logical lookup operations. |
* client.lookup.batch-timeout: Maximum wait time for an underfilled compatible batch.
* client.lookup.max-batch-size: Maximum number of logical lookups in each final RPC.
* client.lookup.max-inflight-requests : Maximum number of unacknowledged lookup RPCs across the client.
* client.lookup.max-inflight-requests-per-server: Maximum number of unacknowledged RPCs containing a given server.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the client.lookup.max-inflight-requests configuration and the existing client.lookup.max-inflight-requests-per-server behavior described in the issue, then compare the related writer.max-inflight-requests-per-bucket setting. Verify how lookup requests are queued and dispatched. Done means requests beyond the per-bucket limit remain queued while other buckets can still dispatch, with configuration and tests covering the new option.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100