Adding new SMT to concatenate key with provided delimiter
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
## Feature request or enhancement
### Which use case/requirement will be addressed by the proposed feature?
In CDC pipelines using Debezium with **compacted topics**, key byte stability is critical.
Currently, Debezium emits structured keys (`Struct`) reflecting the database primary key. When using Avro or JSON converters, the final serialized key bytes may change due to:
* Schema evolution
* Field reordering
* Converter behavior
Kafka compares **raw key bytes**, not logical equality. If serialized bytes change:
* Compaction may fail to remove older records
* Tombstones may not match previous records
* Downstream upsert consumers may produce duplicates
* Partition routing may become unstable
We experienced this in production when using Avro keys with compacted topics.
---
## Proposed enhancement
Provide a Debezium SMT that:
* Concatenates selected key fields (typically PK fields)
* Produces a deterministic `String` key
* Uses a configurable delimiter
* Ensures stable field order
Example:
Database PK:
```sql
PRIMARY KEY (currency, isin, stock_exchange)
```
Structured key → transformed key:
```
CHF|CH1234567890|SIX
```
This ensures:
* Byte-level determinism
* Converter independence
* Stable partitioning
* Safe compaction behavior
---
## Implementation ideas (optional)
Config example:
```properties
transforms=keyConcat
transforms.keyConcat.type=io.debezium.transforms.KeyConcatenation
transforms.keyConcat.fields=currency,isin,stock_exchange
transforms.keyConcat.delimiter=|
```
Options:
* `fields` (default: all key fields in schema order)
* `delimiter` (default: `|`)
* `null.handling` (fail / empty / literal)
Output key schema: `Schema.STRING_SCHEMA`.
We have implemented internally but would be happy to contribute to dbz open source if people are interested.
Contributor guide
Assessment
This issue has not been assessed yet.