[client] Implement adaptive fetch rate control for LogScanner to reduce overhead on partitioned tables
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
For partitioned tables (e.g., hourly partitions with 256 buckets, retaining 24 hours of data), a streaming LogScanner subscribes to all partitions, resulting in 256 × 24 = 6144 buckets. However, only the latest partition's 256 buckets actively receive data. The current LogFetcher implementation treats all buckets equally, sending fetch requests to every bucket each round regardless of whether they have data. This causes unnecessary CPU usage, network overhead, and wasted fetch requests on inactive partitions.
### Solution
_No response_
### Anything else?
_No response_
### Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.