[Feature] Support branch in paimon
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Motivation
Support branch for tables in paimon. In data streaming process there may be data errors and other issues, and we need to correct the data in the flow. This situation is very common and important. However, in this process, we do not want to affect existing data processing to avoid impact on users, we need to create a new data streaming process and wait for it to catch up with the data and replace the original data streaming process. The main operations can be divided into the following steps:
1. Create a replica table based on the specified tag/snapshot of upstream and downstream Paimon Tables
3. Resubmit all streaming jobs, incremental or full recovery starting from the specified offset
After discussed with @SteNicholas , we think we need to support branch in Paimon. Then we could create replica tables to avoid coping all data from specified table and increase storage space. cc @JingsongLi
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Begin by locating Paimon's table and streaming-job APIs that would own branch creation, then clarify the required tag or snapshot behavior and replica-table workflow; done means branches can support the described correction process without copying all table data.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100