apache / apache/paimon

[Feature] Support branch in paimon

Open
#1,795 5 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

Support branch for tables in paimon. In data streaming process there may be data errors and other issues, and we need to correct the data in the flow. This situation is very common and important. However, in this process, we do not want to affect existing data processing to avoid impact on users, we need to create a new data streaming process and wait for it to catch up with the data and replace the original data streaming process. The main operations can be divided into the following steps:
1. Create a replica table based on the specified tag/snapshot of upstream and downstream Paimon Tables
3. Resubmit all streaming jobs, incremental or full recovery starting from the specified offset

After discussed with @SteNicholas , we think we need to support branch in Paimon. Then we could create replica tables to avoid coping all data from specified table and increase storage space. cc @JingsongLi

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Begin by locating Paimon's table and streaming-job APIs that would own branch creation, then clarify the required tag or snapshot behavior and replica-table workflow; done means branches can support the described correction process without copying all table data.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.