cockroachdb / cockroachdb/cockroach
cdc: support different formats for key and value encoding
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
CDC currently only support emitting the entire message in a specific format using `WITH format=...`. But some customers would like to encode the key and values in different formats since this is how their downstream apps use the keys and values.
**Describe the solution you'd like**
Naively thinking about this, we could allow this customization and pass a different encoder or decoder based on the configurations.
https://github.com/cockroachdb/cockroach/blob/1b4f26ed1210e0d5924e9626629737a595c54118/pkg/ccl/changefeedccl/encoder.go#L23-L37 and here https://github.com/cockroachdb/cockroach/blob/c7264978e40967bb73d24667c2574c23a83ea47c/pkg/ccl/changefeedccl/event_processing.go#L428-L435.
**Additional context**
This feature is a customer request (see more details in https://github.com/cockroachlabs/support/issues/2794).
Jira issue: CRDB-35637
Contributor guide
Research direction
Start by reading the encoder definitions in pkg/ccl/changefeedccl/encoder.go around lines 23-37 and the event processing logic in pkg/ccl/changefeedccl/event_processing.go around lines 428-435. Trace how the WITH format configuration selects encoding, then determine the configuration and encoder/decoder boundaries needed to support separate key and value formats. Done means CDC can emit keys and values using independently selected formats while preserving existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100