opensearch-project / opensearch-project/data-prepper

Support Kafka as a Sink

Open
#1,986 7 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

plugin - sink Roadmap:Observability/Log Analytics
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.

Some pipeline authors want to send events from Data Prepper into Kafka. This can allow them to use an existing Kafka stream or send data to other destinations via Kafka.

Describe the solution you'd like

Create a Kafka sink to send events to Kafka. It should batch these events when sending to Kafka.

sink:
- kafka:
    bootstrap_servers:
      - localhost:9092
    topic: my-topic

The configuration should also support many of the producer configs. It should certainly support the configurations which the Kafka client will handle completely and do not require anything from Data Prepper beyond passing the configurations.

The Kafka producer client supports sending to multiple topics. If there is value in making topics dynamic, the configuration could support a parameterized topic. For example, topic: my-topic:${/type}.

For connections, this plugin should use the existing mechanisms in Data Prepper for getting SSL certificates, such as from the file, S3, or ACM.

Additional context

https://kafka.apache.org/33/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the KafkaProducer documentation linked in the issue and inspect Data Prepper's existing mechanisms for SSL certificates from files, S3, or ACM. Define the sink's configuration and batching behavior, including which producer configs can be passed through and whether parameterized topics are needed; done means the requested Kafka output behavior is supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.