opensearch-project / opensearch-project/data-prepper
[BUG]: No back pressure handling, if OpenSearch is temporary overloaded
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 355
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
When OpenSearch cluster is overloaded, DataPrepper instead of generating the warning and waiting exponentially, attempting to deliver the current payload in the infinite loop, which causing more OpenSearch overload.
The log records have also influence to the overload in case, if the target OpenSearch cluster is using for storing logs as well.
To Reproduce
Steps to reproduce the behavior:
Lower OpenSearch cluster to best possible minimum and start sending huge amount of data from DataPrepper (OTLP metrics and traces) and also have some log processor active to feeding the same OpenSearch cluster (Logstash, FluentD, ...).
Expected behavior
When DataPrepper got reject from OpenSearch, it should waiting some time, before attempting to retrying delivery.
Screenshots
Environment (please complete the following information):
- Kubernetes: 1.34
- Version DataPrepper (official Docker image): 2.7.0
- Version OpenSearch (official Docker image): 2.16
Additional context
As you can see from screenshot, the timestamp of the log records are basically same, so DataPrepper not attempting to wait and only crashing to overloaded OpenSearch cluster.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure with a minimally sized OpenSearch cluster while sending large OTLP metrics and traces and feeding logs to the same cluster. Trace DataPrepper's OpenSearch delivery path and its handling of rejected payloads. Done means rejected deliveries produce a warning and wait with exponential backoff before retrying, without continuously worsening the overload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kubernetes
- Domain
- backend, data-engineering, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100