[tiering] Support reporting watermark to Paimon snapshot
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
Currently, Fluss lake tiering can write data into Paimon, but the committed Paimon snapshots do not carry the tiering watermark. This makes it harder to understand the event-time progress of data synchronized from Fluss to the lake table.
Paimon snapshot watermark can be used to better track the synchronization progress of lake tables. By reporting the watermark during tiering commits, users and downstream systems can inspect Paimon snapshots to know how far the lake table has caught up in event time.
### Solution
Propagate watermark information through the lake tiering pipeline and report it when committing to Paimon.
This includes extracting watermarks from Fluss records, carrying them for each split, aggregating them during tiering commit, and writing the final watermark into the committed Paimon snapshot.
### Anything else?
_No response_
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named. Trace the lake tiering pipeline from Fluss record watermark extraction through per-split handling and commit aggregation to the Paimon snapshot commit; done means committed Paimon snapshots contain the aggregated tiering watermark.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100