opensearch-project / opensearch-project/data-prepper

[Feature] Synchronous Buffer

Open
#5,712 0 comments 1 reaction 1 assignee View on GitHub

@MohammedAghil is already working on this.

Since May 27, 2025.

Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
At present, The buffer implementations provide an asynchronous behavior, but there is a need to have a synchronous mechanism where the source thread waits until the events are published to the sink. This allows the source to ensure that the events have been processed.

A similar functionality has been done in Zero Buffer #5415 which allows the source thread to execute the processors and then publish it to sink. But it does not leverage the existing process workers which are available.

Describe the solution you'd like

  • Implement a synchronous buffer which will block the source thread once it writes events to the synchronous buffer until those events has been published to sink.
  • The worker threads will asynchronously read the events from the synchronous buffer and process these events and publish them to sink (async).
  • Once these events are processed and published to sink, the buffer unblocks the source thread.

Additional context
This aims to utilize all the process workers while also ensuring synchronous flow between source and sink.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.