debezium / debezium/dbz

JsonSerde overwrites configuration when used with Spring Cloud Stream [DBZ-3749]

Open
#485 0 comments 0 reactions 0 assignees View on GitHub
component/core-library migrated-from-jira type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-3749](https://issues.redhat.com/browse/DBZ-3749)

When debezium is used with Spring Cloud Stream and Debezium's JsonSerde is used, Serde's configuration get overwritten.

[https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/blob/f7537e795e6c1833405a56e6e7ce86e84612799e/spring-cloud-stream-binder-kafka-streams/src/main/java/org/springframework/cloud/stream/binder/kafka/streams/KeyValueSerdeResolver.java#L405-L426]

Here line 425 calls configure method on already configured Serde with different set of properties.
{code:java}
valueSerde.configure(this.streamConfigGlobalProperties, false);
{code}
[https://github.com/debezium/debezium/blob/5b757372d82f79d314f0392b516cf63245fc72a9/debezium-core/src/main/java/io/debezium/serde/json/JsonSerde.java#L54]

JsonSerde on line 54,  discard old config and overwrites it with new config. Ideally it should merge old config with new one while maintaining precedence.

In my case *from.field* was removed from config when Spring Cloud Stream adds global configurations to existing Serde.

I worked around this issue by extending JsonSerde in following way. This may not be perfect but solves my issue for time being.
{code:java}
public class DebeziumJsonSerde extends JsonSerde {

private final Map configs = new HashMap<>();

public DebeziumJsonSerde(Class type) {
super(type);
}

@Override
public void configure(final Map configs, boolean isKey) {
// First store config to a local map and then pass it on to the super method.
// Everytime configure is called keep appending new config to old one and call super method.
this.configs.putAll(configs);
super.configure(this.configs, isKey);
}
}
{code}

Contributor guide

Open the contributing guide

Research direction

Start in debezium-core/src/main/java/io/debezium/serde/json/JsonSerde.java at configure, then compare it with Spring Cloud Stream's KeyValueSerdeResolver.java lines 405-426. Trace the repeated configure calls and verify that an existing setting such as from.field is retained with the intended precedence. Done means JsonSerde no longer loses earlier configuration when global properties are applied.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.