apache / apache/fluss

[tiering] Tiering Job Performance: Read-Write Pipeline Optimization

Open
#2,915 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

# Problem Description
When running a Tiering job with high write throughput, the data synchronization cannot keep up with the write speed. The root cause analysis reveals two main issues:

1. Parallelism is bounded by bucket count - Tiering job parallelism is 1:1 mapped to bucket count, limiting scalability
2. Read and write operations cannot be pipelined - Reading from Fluss and writing to Paimon are executed sequentially, resulting in low CPU utilization

# Root Cause Analysis

1. Split Granularity Equals Bucket Granularity: Each split covers exactly one bucket, which limits the maximum parallelism.
2. Sequential Read-Write Pattern: The current implementation reads from Fluss and writes to Paimon synchronously.

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the Tiering job implementation, especially the split handling and the code that reads from Fluss and writes to Paimon. Trace how bucket-to-split mapping sets parallelism and where read and write operations are serialized. Done means the job can scale beyond bucket-count parallelism and improve read-write throughput without losing synchronization correctness.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.