apache / apache/fluss

[client] Isolate historical lookup in-flight requests

Open
#3,766 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

Normal and historical lookup requests currently share the `maxInFlightReuqestsSemaphore` in `LookupSender`. This keeps the initial historical lookup implementation simple, but lake lookups can be much slower than normal lookups. A burst of historical requests can therefore consume all in-flight permits and prevent normal lookup requests from being sent until lake requests complete.

FIP-28 proposes separating the capacity available to historical lookups through `client.lookup.historical-inflight-ratio`, with historical requests capped at a small portion of the overall lookup concurrency. The initial implementation intentionally omitted this configuration after discussion because historical lookups were expected to be infrequent.

We should revisit this assumption and isolate or reserve in-flight capacity for historical lookups if shared permits can cause normal lookup starvation. The implementation should:

- prevent historical lookup traffic from consuming all normal lookup permits;
- preserve the existing overall lookup in-flight limit;
- define validation and rounding behavior for small limits and ratio values;
- add coverage showing that normal lookups can proceed while historical lookups are slow.

### Willingness to contribute

- [ ] I am willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with LookupSender and the existing maxInFlightReuqestsSemaphore, then read FIP-28 for the proposed historical-inflight-ratio behavior. Trace how normal and historical lookups acquire permits, define validation and rounding for small limits and ratios, and add coverage proving normal lookups proceed while historical requests are slow.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.