apache / apache/fluss

[client] Implement adaptive fetch rate control for LogScanner to reduce overhead on partitioned tables

Open
#3,006 0 comments 0 reactions 1 assignee Claimed by @swuferhong View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

For partitioned tables (e.g., hourly partitions with 256 buckets, retaining 24 hours of data), a streaming LogScanner subscribes to all partitions, resulting in 256 × 24 = 6144 buckets. However, only the latest partition's 256 buckets actively receive data. The current LogFetcher implementation treats all buckets equally, sending fetch requests to every bucket each round regardless of whether they have data. This causes unnecessary CPU usage, network overhead, and wasted fetch requests on inactive partitions.

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.