[flink] Respect low filesystem rate limits in remove_orphan_files
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
0.9.0 (latest release)
### Please describe the bug 🐞
remove_orphan_files assigns each scan subtask a minimum filesystem operation rate of 1/s. When the scan parallelism exceeds the configured total rate, this increases the aggregate rate above the requested limit.
For example, a total rate of 3 operations/s with parallelism 8 results in an aggregate target of 8 operations/s.
### Solution
Allow fractional per-subtask rates by dividing the configured total rate by the scan parallelism, while retaining the existing best-effort rate-limiting semantics.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate the remove_orphan_files implementation and trace how the configured total filesystem rate is assigned across scan subtasks. Check the existing rate-limiting tests or add coverage for a total rate below the scan parallelism; done means fractional per-subtask rates preserve the requested aggregate best-effort limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100