apache / apache/beam

Coder information lost in Kafka Read

Open
#21,332 0 comments 0 reactions 0 assignees View on GitHub
bug io java kafka P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When upgrading from 2.29.0 to 2.36.0 our Kafka Read transform broke.
While debugging, I saw that information of the key and value coders is lost after [this statement in KafkaIO](https://github.com/apache/beam/blob/v2.36.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1459):
```

return output.apply(readTransform).setCoder(KafkaRecordCoder.of(keyCoder, valueCoder));

```

The issue seems to be that `output.apply` already checks for the presence of key and value coders and ends up trying to infer them with the help of [LocalDeserializerProvider](https://github.com/apache/beam/blob/v2.36.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/LocalDeserializerProvider.java), so the process never arrives at `{}setCoder{`}.
In our case this inference fails since we implement the `Deserializer` interface in a super class of the instance passed as the deserializer (I assume it would also fail afterwards since we don't register our coders with the registry).

This was not yet broken on 2.29.0, so all versions after that could be affected.

Imported from Jira [BEAM-13924](https://issues.apache.org/jira/browse/BEAM-13924). Original Jira may contain additional context.
Reported by: jgrabber.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.