Feature: Add optional commit commit to kafka in s3 plugin
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 504
- Forks
- 263
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 5
Description
Issue
Current implementation: file plugin commits event after written to file, s3 plugin writes zip files to storage.
Desired implementation: file plugin commits event after written to file, s3 plugin writes zip files to storage and commits metadata of upload to kafka.
Feature must be optional.
Solution
Add optional embedding kafka output plugin in s3 plugin and add commitment logic.
Afterwards commits to other sources will be provided.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing file plugin, S3 plugin, and Kafka output plugin implementations to understand their commit behavior and configuration. Add an optional Kafka output path to the S3 plugin, then verify that upload metadata is committed to Kafka only when enabled and after the zip file is stored.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kafka
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100