opensearch-project / opensearch-project/data-prepper
Implement lumberjack protocol for input/sink
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
The forward protocol is used for filebeat/logstash setups to forward logs between instances. This is very useful for larger architectures. It would be nice to implement this for data prepper as well so data prepper can be used as an intermediate or endpoint before sending the data to Opensearch and can fully replace logstash.
Describe the solution you'd like
Implement input/sink
Describe alternatives you've considered (Optional)
http - overhead might be an issue
Additional context
none
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing forward protocol implementation and the input and sink extension points in Data Prepper. Define the Lumberjack protocol scope, then identify the required input and sink behavior and corresponding tests; done means Data Prepper can reliably act as an intermediate or endpoint in the described Filebeat/Logstash-style setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100