[Feature] Kafka CDC with debezium/ canal json support lacking of before in update
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
@yuzelin
由于我们采用整库同步,单个 topic 里存在多个表,CDC采集侧也不受我们控制,受限于整体的数据量大,其中仅有极少数CDC的update数据有before前项。
我们惊奇发现 [Apache Paimon changelog producer](https://paimon.apache.org/docs/master/primary-key-table/changelog-producer/),因此想要同步至 Paimon ODS 中采用 lookup 方式来产生正确的 CDC给后续链路提供支持。但在测试过程中发现若 Kafka CDC 对于 update 数据中缺少 before 的数据会报错。
因此希望Kafka CDC 整库同步可以适配 缺少前项的 update,或者提供一种解决方案,我们将收到的 Kafka CDC 数据进行调整,再打到 自己的 Kafka 队列中进行消费。
---
English translation
Due to our adoption of whole-database synchronization, there are multiple tables within a single topic. We do not have control over the CDC collection side, and due to the large overall data volume, only a very small number of CDC update data contain the before field.
We were surprised to discover the [Apache Paimon changelog producer](https://paimon.apache.org/docs/master/primary-key-table/changelog-producer/), so we want to use lookup in synchronizing with Paimon ODS to generate correct CDCs for subsequent links. However, during testing, we found that if Kafka CDC lacks before data in update records, an error will occur.
Therefore, we hope that Kafka CDC's whole-database synchronization can adapt to updates lacking before fields or provide a solution where we adjust the received Kafka CDC data before sending it to our own Kafka queue for consumption.
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Kafka CDC whole-database synchronization failure with an update record that lacks the before field. Trace the Kafka CDC ingestion path and compare it with the changelog producer lookup behavior described in the issue. Done means an agreed, tested behavior for such updates or a documented solution for preprocessing them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100