apache / apache/paimon

[Feature] Kafka CDC with debezium/ canal json support lacking of before in update

Open
#3,176 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

@yuzelin

由于我们采用整库同步,单个 topic 里存在多个表,CDC采集侧也不受我们控制,受限于整体的数据量大,其中仅有极少数CDC的update数据有before前项。

我们惊奇发现 [Apache Paimon changelog producer](https://paimon.apache.org/docs/master/primary-key-table/changelog-producer/),因此想要同步至 Paimon ODS 中采用 lookup 方式来产生正确的 CDC给后续链路提供支持。但在测试过程中发现若 Kafka CDC 对于 update 数据中缺少 before 的数据会报错。

因此希望Kafka CDC 整库同步可以适配 缺少前项的 update,或者提供一种解决方案,我们将收到的 Kafka CDC 数据进行调整,再打到 自己的 Kafka 队列中进行消费。

---
English translation

Due to our adoption of whole-database synchronization, there are multiple tables within a single topic. We do not have control over the CDC collection side, and due to the large overall data volume, only a very small number of CDC update data contain the before field.

We were surprised to discover the [Apache Paimon changelog producer](https://paimon.apache.org/docs/master/primary-key-table/changelog-producer/), so we want to use lookup in synchronizing with Paimon ODS to generate correct CDCs for subsequent links. However, during testing, we found that if Kafka CDC lacks before data in update records, an error will occur.

Therefore, we hope that Kafka CDC's whole-database synchronization can adapt to updates lacking before fields or provide a solution where we adjust the received Kafka CDC data before sending it to our own Kafka queue for consumption.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Kafka CDC whole-database synchronization failure with an update record that lacks the before field. Trace the Kafka CDC ingestion path and compare it with the changelog producer lookup behavior described in the issue. Done means an agreed, tested behavior for such updates or a documented solution for preprocessing them.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.