apache / apache/paimon

[Feature] [cdc] Support merge_shards in kafka/pulsar database synchronization

Open
#3,352 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

binlog data in the same topic may contain data with the same table name in different database. Provide the merge_shards parameter like Mysql-cdc to avoid table name conflicts.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the existing MySQL CDC implementation of merge_shards and the Kafka/Pulsar database synchronization entry points. Compare how the parameter avoids table-name conflicts across databases, then define the affected synchronization paths and tests; done means merge_shards works for the requested Kafka and Pulsar scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.