digitalocean / digitalocean/firebolt
Support for reading / writing kafka message headers
- Dominant language
- Go
- Stars
- 719
- Forks
- 50
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
Nice work on this project - it looks really cool!
My shop is looking at using this as the base for a golang stream processor. We have existing SPs which implement our message format / conventions, one of which is that we use kafka headers for message metadata (correlation_id, timestamp, ser/des details, etc).
I see that the provided `KafkaConsumer` passes on the received message's body to the next node, but does not pass headers.
- https://github.com/digitalocean/firebolt/blob/main/node/kafkaconsumer/kafkaconsumer.go#L221
- https://github.com/confluentinc/confluent-kafka-go/blob/master/kafka/message.go#L72-L80
Similarly, the provided `KafkaProducer` does not provide a way to assign/write headers (that I can see).
- https://github.com/digitalocean/firebolt/blob/main/node/kafkaproducer/kafkaproducer.go#L108-L111
Do you have any guidance on the best way to accomplish this?
One option seems to be to copy `KafkaConsumer` and `KafkaProducer` and modify them to do what we want. This seems inelegant, since there would be a ton of functionality in those that we would not be touching, but would be accepting a maintenance burden for.
Or maybe (hopefully?) I've missed something and there is some header functionality already available?
Thanks,
Dan
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.