opensearch-project / opensearch-project/data-prepper
[BUG] OpensearchSink not flushing remaining request on graceful shutdown
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 355
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
On graceful shutdown request opensearch sink should try to flush the remaining request at hand before cleaning up resources.
There is data loss because opensearch sink not flushing the remaining request on shutdown phase.
To Reproduce
Steps to reproduce the behavior:
- start 2 data prepper instances say data_prepper_1, data_prepper_2 with opensearch sink configured
- Upon handling sufficient data, perform restart on data_prepper_1
- Compare the output data of the above two instances
Expected behavior
Both data_prepper_1, data_prepper_2 output event count should be same.
Screenshots
Environment (please complete the following information):
- OS: [e.g. Ubuntu 20.04 LTS]
- Version [e.g. 22]
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the OpenSearchSink graceful-shutdown and cleanup path, then reproduce the restart scenario with two Data Prepper instances and compare event counts. Done means the remaining request is flushed before resources are cleaned up and both instances retain the same event count.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100