apache / apache/fluss

[tiering] Support reporting watermark to Paimon snapshot

Open
#3,419 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

Currently, Fluss lake tiering can write data into Paimon, but the committed Paimon snapshots do not carry the tiering watermark. This makes it harder to understand the event-time progress of data synchronized from Fluss to the lake table.

Paimon snapshot watermark can be used to better track the synchronization progress of lake tables. By reporting the watermark during tiering commits, users and downstream systems can inspect Paimon snapshots to know how far the lake table has caught up in event time.

### Solution

Propagate watermark information through the lake tiering pipeline and report it when committing to Paimon.

This includes extracting watermarks from Fluss records, carrying them for each split, aggregating them during tiering commit, and writing the final watermark into the committed Paimon snapshot.

### Anything else?

_No response_

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Trace the lake tiering pipeline from Fluss record watermark extraction through per-split handling and commit aggregation to the Paimon snapshot commit; done means committed Paimon snapshots contain the aggregated tiering watermark.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.