apache / apache/paimon

[Feature] Cdc support new synchronization mode

Open
#3,786 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

paimon cdc 支持divided和 combined 同步模式,对大量表同步场景,两种模式都有一定局限性。一般业务数据库大表很少,可能top5内,小表比较多,两种模式都不合适。
建议一种新模式,支持为大表设置独立sink,剩余表在一个sink中,可以为每个sink 设置并发度。

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named in the issue, so first clarify the proposed CDC synchronization mode and how it differs from divided and combined modes. Done should include a defined way to assign large tables to independent sinks, group remaining tables into one sink, and configure concurrency per sink.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.