opensearch-project / opensearch-project/data-prepper
Provide a pipe processor for arbitrary execution of input and output
@dlvenable is already working on this.
Since Jan 7, 2025.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
Some use-cases, such as GenAI, could be solved through running external programs. For example, to run data from Data Prepper into a Python program running LangChain, we could send events in and then enrich those events with the output from the program.
Describe the solution you'd like
Create a pipe processor. This processor will call an external command with arguments to modify events.
It operates similar to a Linux pipe (really two pipes). It would put each event on a newline as JSON into the program's stdin. Then it would read from stdout to get changes to the event to merge in.
Conceptually it is similar to data-prepper | command | data-prepper
processor:
- pipe:
command: python3
arguments:
- /path/to/my/script
Describe alternatives you've considered (Optional)
The AWS Lambda processor is quite similar. However, this requires the use of AWS Lambda and does not work for fully open-source use-cases.
I also considered creating both a Python processor and a LangChain processor. But, a general pipe processor would be more generic. And creating processors for these specific tools would not have much value if they don't already include the dependencies, but then we may get into some dependency issues.
Additional context
N/A
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.