opensearch-project / opensearch-project/data-prepper
Documentation on how to configure pipelines with disk based buffering
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
https://opensearch.org/docs/2.4/data-prepper/pipelines/configuration/buffers/buffers/
Buffers store data as it passes through the pipeline. If you implement a custom buffer, it can be memory based, which provides better performance, or disk based, which is larger in size.
The above said documentation states that "disk based" buffering can be configured instead of a memory based "bounded_blocking" one. But I don't see the parameter names or ways to configure it
Can someone pls give an example where pipeline parts like "entry-pipeline", "raw-pipeline" and "service-pipeline" are configured with disk based buffering.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the buffers documentation at the linked URL and inspect the Data Prepper pipeline configuration for the disk-based buffer parameter names. Document how entry-pipeline, raw-pipeline, and service-pipeline use disk-based buffering, including a configuration example; done means the setup is explicit and reproducible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100