[Feature] Trigger savepoint automatically if the aligned snapshots in sources are created by savepoint
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Motivation
Related to #1635 and #1486
When there are ETL jobs and Paimon Tables associated upstream and downstream, the upstream job creates a tag through savepoint, and downstream jobs consume the tag data and align it, they could automatically trigger a savepoint instead of checkpoint. For example, there are tables TableA, TableB and TableC, Job1 processes data from TableA to TableB, Job2 processes TableB to TableC
TableA --> Job1 --> TableB --> Job2 --> TableC
When Job1 save a tag named tagB1 in TableB by savepoint, Job2 will trigger a savepoint for tagB1 and save a tag named tagC1.
When we need to resume the above data processing, we can create a new Job1' and Job2' from the above savenpoints and tags in TableB and TableC to incremental recover data processing. cc @liming30
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how aligned snapshots distinguish checkpoints from savepoints and how upstream tags are consumed across chained jobs. Define the behavior for propagating a savepoint from TableB to TableC, including recovery through the resulting tags; the issue names no files or tests, so the relevant implementation and validation points must first be located.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100