debezium / debezium/dbz

Generify ExtractNewRecordState to support pluggable Messaging systems [DBZ-1608]

Open
#269 0 comments 0 reactions 0 assignees View on GitHub
component/core-library migrated-from-jira type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-1608](https://issues.redhat.com/browse/DBZ-1608)

Right now [ExtractNewRecordState.java|https://github.com/debezium/debezium/blob/master/debezium-core/src/main/java/io/debezium/transforms/ExtractNewRecordState.java] is tightly coupled and implemented based on Kafka. We are using Debezium with Apache Pulsar for CDC but one missing feature is the Record Transformation in producer side before publishing to the topic.

As per the Apache Pulsar [Transformation support in Github issue|https://github.com/apache/pulsar/issues/4672#issuecomment-509877837], tight coupling nature of ExtractNewRecordState with Kafka is a blocker. Instead of writing our implementation of ExtractNewRecordState for Pulsar, it would be best to Generify the ExtractNewRecordState class in debezium to support any messaging system (which should be configurable at connector setup).

PS: The comment in the provided Github issue link is by official maintainer / contributor of Apache Pulsar. I am just an end user of Apache Pulsar.

Contributor guide

Open the contributing guide

Research direction

Start by reading debezium-core/src/main/java/io/debezium/transforms/ExtractNewRecordState.java and the linked Apache Pulsar transformation discussion. Identify where the transform is coupled to Kafka and how connector setup would need to select messaging-system behavior. The issue is done when the transform supports the requested pluggable messaging systems without requiring a separate Pulsar implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.