apache / apache/fluss

[tiering] Slow manifest compaction during lake commit blocks other tables' tiering progress

Open
#3,259 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

#### Problem
The TieringCommitOperator currently runs with parallelism=1 and maxParallelism=1. All table commits are processed sequentially in a single thread. When a table's lake commit triggers manifest compaction (e.g., in Paimon), it can take a significant amount of time, during which all other tables' commits are blocked.This causes data freshness degradation for unrelated tables, potentially triggering alerts even though those tables have data ready to commit.

#### Current Behavior
The commit flow in TieringCommitOperator.processElement() is synchronous:
1. Collect all bucket write results for a table
2. Call LakeCommitter.toCommittable()
3. Call LakeCommitter.commit() — this is the bottleneck, as it may trigger manifest compaction
4. Call FlussTableLakeSnapshotCommitter.commit() to notify Fluss

Since the operator is single-threaded with parallelism=1, step 3 for one table blocks steps 1-4 for all other tables.

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in TieringCommitOperator.processElement() and trace the synchronous calls to LakeCommitter.toCommittable(), LakeCommitter.commit(), and FlussTableLakeSnapshotCommitter.commit(). Determine how the single-threaded flow blocks unrelated tables, then verify that a slow lake commit no longer prevents other tables from making tiering progress.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.