[client] Isolate historical lookup in-flight requests
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
Normal and historical lookup requests currently share the `maxInFlightReuqestsSemaphore` in `LookupSender`. This keeps the initial historical lookup implementation simple, but lake lookups can be much slower than normal lookups. A burst of historical requests can therefore consume all in-flight permits and prevent normal lookup requests from being sent until lake requests complete.
FIP-28 proposes separating the capacity available to historical lookups through `client.lookup.historical-inflight-ratio`, with historical requests capped at a small portion of the overall lookup concurrency. The initial implementation intentionally omitted this configuration after discussion because historical lookups were expected to be infrequent.
We should revisit this assumption and isolate or reserve in-flight capacity for historical lookups if shared permits can cause normal lookup starvation. The implementation should:
- prevent historical lookup traffic from consuming all normal lookup permits;
- preserve the existing overall lookup in-flight limit;
- define validation and rounding behavior for small limits and ratio values;
- add coverage showing that normal lookups can proceed while historical lookups are slow.
### Willingness to contribute
- [ ] I am willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with LookupSender and the existing maxInFlightReuqestsSemaphore, then read FIP-28 for the proposed historical-inflight-ratio behavior. Trace how normal and historical lookups acquire permits, define validation and rounding for small limits and ratios, and add coverage proving normal lookups proceed while historical requests are slow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100