apache / apache/fluss

[Client] Add client.lookup.max-inflight-requests-per-bucket to restrict lookup requests per server

Open
#4,043 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

0.9.0 (latest release)

### Please describe the bug 🐞
`client.lookup.max-inflight-requests` currently provides only a client-wide RPC concurrency limit.

During leader failover, such as a rolling upgrade, requests sent to an old or temporarily unavailable leader may remain incomplete until the connection fails or the request times out.

If many lookups target the same affected bucket, their RPCs may consume most or all global in-flight permits:

`client.lookup.max-inflight-server` (which is same as `writer.max-inflight-requests-per-bucket`

Once the global permits are exhausted, the lookup sender blocks before sending additional RPCs. Requests for healthy buckets and healthy TabletServers can no longer be dispatched, even though they are unrelated to the failed bucket.

To isolate this failure, the lookup client needs an additional per-bucket limit:

`client.lookup.max-inflight-requests-per-bucket`

Once one bucket reaches this limit, further requests for that bucket should remain queued, while requests for other buckets remain eligible for dispatch.

This is specifically per-bucket isolation. Multiple affected buckets on the same failed TabletServer can still collectively consume global permits; the option is not a per-node circuit breaker.

### Solution

* client.lookup.queue-size : Maximum number of admitted logical lookup operations. |
* client.lookup.batch-timeout: Maximum wait time for an underfilled compatible batch.
* client.lookup.max-batch-size: Maximum number of logical lookups in each final RPC.
* client.lookup.max-inflight-requests : Maximum number of unacknowledged lookup RPCs across the client.
* client.lookup.max-inflight-requests-per-server: Maximum number of unacknowledged RPCs containing a given server.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the client.lookup.max-inflight-requests configuration and the existing client.lookup.max-inflight-requests-per-server behavior described in the issue, then compare the related writer.max-inflight-requests-per-bucket setting. Verify how lookup requests are queued and dispatched. Done means requests beyond the per-bucket limit remain queued while other buckets can still dispatch, with configuration and tests covering the new option.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.