apache / apache/paimon

[Feature] Trigger savepoint automatically if the aligned snapshots in sources are created by savepoint

Open
#1,834 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

Related to #1635 and #1486
When there are ETL jobs and Paimon Tables associated upstream and downstream, the upstream job creates a tag through savepoint, and downstream jobs consume the tag data and align it, they could automatically trigger a savepoint instead of checkpoint. For example, there are tables TableA, TableB and TableC, Job1 processes data from TableA to TableB, Job2 processes TableB to TableC
TableA --> Job1 --> TableB --> Job2 --> TableC
When Job1 save a tag named tagB1 in TableB by savepoint, Job2 will trigger a savepoint for tagB1 and save a tag named tagC1.

When we need to resume the above data processing, we can create a new Job1' and Job2' from the above savenpoints and tags in TableB and TableC to incremental recover data processing. cc @liming30

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how aligned snapshots distinguish checkpoints from savepoints and how upstream tags are consumed across chained jobs. Define the behavior for propagating a savepoint from TableB to TableC, including recovery through the resulting tags; the issue names no files or tests, so the relevant implementation and validation points must first be located.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.