pingcap / pingcap/ticdc

pulsar: support debezium protocol

Open Beginner friendly
#5,056 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

contribution type/enhancement
Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

Is your feature request related to a problem?

The Pulsar sink only accepts canal-json as the message protocol. Users who consume TiCDC change events via Pulsar with Debezium-compatible downstream systems (e.g. Flink CDC, Debezium connectors) have no way to produce standard Debezium-format messages, even though the Debezium codec is already implemented and used by the Kafka sink.

Describe the feature you'd like

Allow protocol=debezium to be set in the Pulsar sink URI:

pulsar://host:6650/topic?protocol=debezium

This requires extending IsPulsarSupportedProtocols() in pkg/config/sink_protocol.go to include ProtocolDebezium. No new codec logic is needed since the Debezium encoder/decoder in pkg/sink/codec/debezium/ is already shared with the Kafka sink via the common codec builder.

Describe alternatives you've considered

Users currently must use canal-json and implement a custom transform layer in their consumer to convert to Debezium format. This adds operational overhead and is error-prone.

Teachability, Documentation, Adoption, Migration Strategy

The Pulsar sink URI parameter documentation should be updated to list debezium as a valid value for the protocol parameter alongside canal-json. No migration is required for existing changefeeds since the default protocol is unchanged.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/config/sink_protocol.go at IsPulsarSupportedProtocols(), then inspect the shared codec handling under pkg/sink/codec/debezium/. Update the Pulsar sink protocol options so protocol=debezium is accepted, and update the Pulsar sink URI documentation to list it alongside canal-json; existing defaults should remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
stream-processing
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.