apache / apache/hop

[Feature Request]: Allow reading of non-string messages with a Kafka Consumer

Open
#2,741 1 comment 0 reactions 0 assignees View on GitHub
awaiting triage P3 Transforms
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 33m
Merged PRs (30d)
204

Description

### What would you like to happen?

Migration: https://issues.apache.org/jira/browse/HOP-3687

For both the regular Kafka Consumer and the Beam Kafka Consumer transforms it would be great to be able to read non-String messages, for example in Avro format.

For the Beam Kafka Consumer I think it would be cool if you could specify the output format as "Avro Record" (in our example) or Serializable, Binary, with String as a default. 

This would allow us to set Kafka consumer option value.deserializer to KafkaAvroDeserializer and get an actual Avro Record as a message.

### Issue Priority

Priority: 3

### Issue Component

Component: Transforms

Contributor guide

Open the contributing guide

Research direction

Start with the regular Kafka Consumer and Beam Kafka Consumer transforms, then trace how their message values are currently handled as Strings and how the value.deserializer option is exposed. Done means both transforms can read non-string values such as Avro records or binary data, while retaining String as the default and allowing the requested output formats.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.