apache / apache/paimon

[Feature] Support sync newly added table in CDC ingestion

Open
#1,181 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

After CdcSyncDatabaseAction job start (for now we only sync a single database), if new table is created in MySQL, then action should sync the new mysql table to paimon. Specifically,

1. The new Paimon table is created upon receiving the create table DDL event in CDC source. Need to respect prefix/suffix.
2. CDC changelog of the newly added table should be successfully written to Paimon
3. Support multiple newly added tables.

### Solution

The draft design is as follows:
1. The new Paimon table is created upon receiving the create table DDL event in CDC source. Need to respect prefix/suffix.

Therefore we need to parse create table statement and utilize Paimon catalog to create the physical table

2. CDC changelog of the newly added table should be successfully written to Paimon

So we need another type of Paimon sink designed to write changelog from newly added tables to their corresponding Paimon tables.

3. Support multiple newly added tables.

Specifically, the new Paimon sink needs to write the data from newly added tables discovered during runtime by managing multiple Paimon tables at the same time.

### Anything else?

To accomplish this feature, it's better to go by several PRs:

1. PoC: implement a first draft to sync newly added table.
2. Implement a multiplexed Paimon sink writer and committer to support multi-table write
3. Implement a multiplexed Paimon compactor to support multi-table compaction
4. End-to-end test against newly added table scenario

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with CdcSyncDatabaseAction and the CDC source handling for create-table DDL events, then review the Paimon catalog and the proposed sink writer, committer, and compactor responsibilities. The work is complete when newly created MySQL tables are created with the required prefix and suffix, their changelogs are written successfully, multiple tables are supported, and the end-to-end scenario is tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
data-engineering, databases, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.