opensearch-project / opensearch-project/data-prepper
[Feature] Synchronous Buffer
@MohammedAghil is already working on this.
Since May 27, 2025.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
At present, The buffer implementations provide an asynchronous behavior, but there is a need to have a synchronous mechanism where the source thread waits until the events are published to the sink. This allows the source to ensure that the events have been processed.
A similar functionality has been done in Zero Buffer #5415 which allows the source thread to execute the processors and then publish it to sink. But it does not leverage the existing process workers which are available.
Describe the solution you'd like
- Implement a synchronous buffer which will block the source thread once it writes events to the synchronous buffer until those events has been published to sink.
- The worker threads will asynchronously read the events from the synchronous buffer and process these events and publish them to sink (async).
- Once these events are processed and published to sink, the buffer unblocks the source thread.
Additional context
This aims to utilize all the process workers while also ensuring synchronous flow between source and sink.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.