apache / apache/fluss

[flink] Respect low filesystem rate limits in remove_orphan_files

Open Beginner friendly
#4,320 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

0.9.0 (latest release)

### Please describe the bug 🐞

remove_orphan_files assigns each scan subtask a minimum filesystem operation rate of 1/s. When the scan parallelism exceeds the configured total rate, this increases the aggregate rate above the requested limit.

For example, a total rate of 3 operations/s with parallelism 8 results in an aggregate target of 8 operations/s.

### Solution

Allow fractional per-subtask rates by dividing the configured total rate by the scan parallelism, while retaining the existing best-effort rate-limiting semantics.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the remove_orphan_files implementation and trace how the configured total filesystem rate is assigned across scan subtasks. Check the existing rate-limiting tests or add coverage for a total rate below the scan parallelism; done means fractional per-subtask rates preserve the requested aggregate best-effort limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.