debezium / debezium/dbz

Adding new SMT to concatenate key with provided delimiter

Open
#1,619 4 comments 0 reactions 0 assignees View on GitHub
component/core-library type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

## Feature request or enhancement

### Which use case/requirement will be addressed by the proposed feature?

In CDC pipelines using Debezium with **compacted topics**, key byte stability is critical.

Currently, Debezium emits structured keys (`Struct`) reflecting the database primary key. When using Avro or JSON converters, the final serialized key bytes may change due to:

* Schema evolution
* Field reordering
* Converter behavior

Kafka compares **raw key bytes**, not logical equality. If serialized bytes change:

* Compaction may fail to remove older records
* Tombstones may not match previous records
* Downstream upsert consumers may produce duplicates
* Partition routing may become unstable

We experienced this in production when using Avro keys with compacted topics.

---

## Proposed enhancement

Provide a Debezium SMT that:

* Concatenates selected key fields (typically PK fields)
* Produces a deterministic `String` key
* Uses a configurable delimiter
* Ensures stable field order

Example:

Database PK:

```sql
PRIMARY KEY (currency, isin, stock_exchange)
```

Structured key → transformed key:

```
CHF|CH1234567890|SIX
```

This ensures:

* Byte-level determinism
* Converter independence
* Stable partitioning
* Safe compaction behavior

---

## Implementation ideas (optional)

Config example:

```properties
transforms=keyConcat
transforms.keyConcat.type=io.debezium.transforms.KeyConcatenation
transforms.keyConcat.fields=currency,isin,stock_exchange
transforms.keyConcat.delimiter=|
```

Options:

* `fields` (default: all key fields in schema order)
* `delimiter` (default: `|`)
* `null.handling` (fail / empty / literal)

Output key schema: `Schema.STRING_SCHEMA`.

We have implemented internally but would be happy to contribute to dbz open source if people are interested.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.