opensearch-project / opensearch-project/data-prepper
Support Kafka as a Sink
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
Some pipeline authors want to send events from Data Prepper into Kafka. This can allow them to use an existing Kafka stream or send data to other destinations via Kafka.
Describe the solution you'd like
Create a Kafka sink to send events to Kafka. It should batch these events when sending to Kafka.
sink:
- kafka:
bootstrap_servers:
- localhost:9092
topic: my-topic
The configuration should also support many of the producer configs. It should certainly support the configurations which the Kafka client will handle completely and do not require anything from Data Prepper beyond passing the configurations.
The Kafka producer client supports sending to multiple topics. If there is value in making topics dynamic, the configuration could support a parameterized topic. For example, topic: my-topic:${/type}.
For connections, this plugin should use the existing mechanisms in Data Prepper for getting SSL certificates, such as from the file, S3, or ACM.
Additional context
https://kafka.apache.org/33/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the KafkaProducer documentation linked in the issue and inspect Data Prepper's existing mechanisms for SSL certificates from files, S3, or ACM. Define the sink's configuration and batching behavior, including which producer configs can be passed through and whether parameterized topics are needed; done means the requested Kafka output behavior is supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100