cockroachdb / cockroachdb/cockroach

cdc: support different formats for key and value encoding

Open
#118,242 2 comments 0 reactions 0 assignees View on GitHub
A-cdc C-enhancement P-3 T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**
CDC currently only support emitting the entire message in a specific format using `WITH format=...`. But some customers would like to encode the key and values in different formats since this is how their downstream apps use the keys and values.

**Describe the solution you'd like**
Naively thinking about this, we could allow this customization and pass a different encoder or decoder based on the configurations.

https://github.com/cockroachdb/cockroach/blob/1b4f26ed1210e0d5924e9626629737a595c54118/pkg/ccl/changefeedccl/encoder.go#L23-L37 and here https://github.com/cockroachdb/cockroach/blob/c7264978e40967bb73d24667c2574c23a83ea47c/pkg/ccl/changefeedccl/event_processing.go#L428-L435.

**Additional context**
This feature is a customer request (see more details in https://github.com/cockroachlabs/support/issues/2794).

Jira issue: CRDB-35637

Contributor guide

Open the contributing guide

Research direction

Start by reading the encoder definitions in pkg/ccl/changefeedccl/encoder.go around lines 23-37 and the event processing logic in pkg/ccl/changefeedccl/event_processing.go around lines 428-435. Trace how the WITH format configuration selects encoding, then determine the configuration and encoder/decoder boundaries needed to support separate key and value formats. Done means CDC can emit keys and values using independently selected formats while preserving existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.