opensearch-project / opensearch-project/data-prepper
Processor scripting language support
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 355
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
Some other logforwarders support flexible scripting, i.e. Fluent-bit supports Lua as an option to perform more complex operations than the pipeline might support. This would enable the user to build complex handling and extend the processing capabilities. We have used this to do more complex conditional logic.
Describe the solution you'd like
Lua/Javascript/whatever language would be really nice to be able to use. It needs to be performant and needs to fit the data prepper architecture.
Describe alternatives you've considered (Optional)
Using fluent-bit itself - but I have found it less reliable than data prepper or logstash in some circumstances.
Additional context
None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by reviewing Data Prepper's processor architecture and the scripting approaches used by Fluent Bit, then determine which language and integration design fit the architecture; done should mean a defined, performant scripting processor with clear support boundaries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, lua
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100