[tiering] Slow manifest compaction during lake commit blocks other tables' tiering progress
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
#### Problem
The TieringCommitOperator currently runs with parallelism=1 and maxParallelism=1. All table commits are processed sequentially in a single thread. When a table's lake commit triggers manifest compaction (e.g., in Paimon), it can take a significant amount of time, during which all other tables' commits are blocked.This causes data freshness degradation for unrelated tables, potentially triggering alerts even though those tables have data ready to commit.
#### Current Behavior
The commit flow in TieringCommitOperator.processElement() is synchronous:
1. Collect all bucket write results for a table
2. Call LakeCommitter.toCommittable()
3. Call LakeCommitter.commit() — this is the bottleneck, as it may trigger manifest compaction
4. Call FlussTableLakeSnapshotCommitter.commit() to notify Fluss
Since the operator is single-threaded with parallelism=1, step 3 for one table blocks steps 1-4 for all other tables.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in TieringCommitOperator.processElement() and trace the synchronous calls to LakeCommitter.toCommittable(), LakeCommitter.commit(), and FlussTableLakeSnapshotCommitter.commit(). Determine how the single-threaded flow blocks unrelated tables, then verify that a slow lake commit no longer prevents other tables from making tiering progress.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100