apache / apache/paimon

[Feature] Support partition watermark notification

Open
#1,047 6 comments 0 reactions 1 assignee Claimed by @SteNicholas View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

It is necessary to know whether the data written to the partition has been completed, so that the downstream data warehouse can trigger batch running

### Solution

1. add watermark timestamp in snapshot meta data when checkppoint
2. write a success file in target partition when data finish in this partition
3. tirgger a notification when data finish in this partition

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.