opensearch-project / opensearch-project/data-prepper
Common Buffer Accumulator
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 355
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
Pull based sources (S3, OpenSearch) collect data in temporary buffer (BufferAccumulator) prior to writing the data to Data Prepper buffer. This code is an exact copy and this will have us supporting the same class in multiple places. We will likely encounter the need for this elsewhere with other pull sources on the road map.
Describe the solution you'd like
A common class that can be leveraged in pull source plugins to temporarily hold data prior to writing to the buffer.
Describe alternatives you've considered (Optional)
- Writing to the buffer directly and eliminating this buffer.
Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the duplicated BufferAccumulator implementations used by the S3 and OpenSearch pull sources, then read PR 2734 for the surrounding design. Determine where a shared pull-source class belongs and verify that both existing users can rely on it without duplicated implementations. Done means the common accumulator is available to pull source plugins and the existing use cases remain covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100