digitalocean / digitalocean/firebolt

Support for reading / writing kafka message headers

Open
#46 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
719
Forks
50
PR merge metrics
No merged PRs in 30d

Description

Hello,

Nice work on this project - it looks really cool!

My shop is looking at using this as the base for a golang stream processor. We have existing SPs which implement our message format / conventions, one of which is that we use kafka headers for message metadata (correlation_id, timestamp, ser/des details, etc).

I see that the provided `KafkaConsumer` passes on the received message's body to the next node, but does not pass headers.
- https://github.com/digitalocean/firebolt/blob/main/node/kafkaconsumer/kafkaconsumer.go#L221
- https://github.com/confluentinc/confluent-kafka-go/blob/master/kafka/message.go#L72-L80

Similarly, the provided `KafkaProducer` does not provide a way to assign/write headers (that I can see).
- https://github.com/digitalocean/firebolt/blob/main/node/kafkaproducer/kafkaproducer.go#L108-L111

Do you have any guidance on the best way to accomplish this?

One option seems to be to copy `KafkaConsumer` and `KafkaProducer` and modify them to do what we want. This seems inelegant, since there would be a ton of functionality in those that we would not be touching, but would be accepting a maintenance burden for.

Or maybe (hopefully?) I've missed something and there is some header functionality already available?

Thanks,
Dan

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.