[Feature Request]: Allow reading of non-string messages with a Kafka Consumer
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 33m
- Merged PRs (30d)
- 204
Description
### What would you like to happen?
Migration: https://issues.apache.org/jira/browse/HOP-3687
For both the regular Kafka Consumer and the Beam Kafka Consumer transforms it would be great to be able to read non-String messages, for example in Avro format.
For the Beam Kafka Consumer I think it would be cool if you could specify the output format as "Avro Record" (in our example) or Serializable, Binary, with String as a default.
This would allow us to set Kafka consumer option value.deserializer to KafkaAvroDeserializer and get an actual Avro Record as a message.
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms
Contributor guide
Research direction
Start with the regular Kafka Consumer and Beam Kafka Consumer transforms, then trace how their message values are currently handled as Strings and how the value.deserializer option is exposed. Done means both transforms can read non-string values such as Avro records or binary data, while retaining String as the default and allowing the requested output formats.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100