JsonSerde overwrites configuration when used with Spring Cloud Stream [DBZ-3749]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-3749](https://issues.redhat.com/browse/DBZ-3749)
When debezium is used with Spring Cloud Stream and Debezium's JsonSerde is used, Serde's configuration get overwritten.
[https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/blob/f7537e795e6c1833405a56e6e7ce86e84612799e/spring-cloud-stream-binder-kafka-streams/src/main/java/org/springframework/cloud/stream/binder/kafka/streams/KeyValueSerdeResolver.java#L405-L426]
Here line 425 calls configure method on already configured Serde with different set of properties.
{code:java}
valueSerde.configure(this.streamConfigGlobalProperties, false);
{code}
[https://github.com/debezium/debezium/blob/5b757372d82f79d314f0392b516cf63245fc72a9/debezium-core/src/main/java/io/debezium/serde/json/JsonSerde.java#L54]
JsonSerde on line 54, discard old config and overwrites it with new config. Ideally it should merge old config with new one while maintaining precedence.
In my case *from.field* was removed from config when Spring Cloud Stream adds global configurations to existing Serde.
I worked around this issue by extending JsonSerde in following way. This may not be perfect but solves my issue for time being.
{code:java}
public class DebeziumJsonSerde extends JsonSerde {
private final Map configs = new HashMap<>();
public DebeziumJsonSerde(Class type) {
super(type);
}
@Override
public void configure(final Map configs, boolean isKey) {
// First store config to a local map and then pass it on to the super method.
// Everytime configure is called keep appending new config to old one and call super method.
this.configs.putAll(configs);
super.configure(this.configs, isKey);
}
}
{code}
Contributor guide
Research direction
Start in debezium-core/src/main/java/io/debezium/serde/json/JsonSerde.java at configure, then compare it with Spring Cloud Stream's KeyValueSerdeResolver.java lines 405-426. Trace the repeated configure calls and verify that an existing setting such as from.field is retained with the intended precedence. Done means JsonSerde no longer loses earlier configuration when global properties are applied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100