opensearch-project / opensearch-project/data-prepper

[BUG]: No back pressure handling, if OpenSearch is temporary overloaded

Open
#6,901 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
374
Forks
355
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
When OpenSearch cluster is overloaded, DataPrepper instead of generating the warning and waiting exponentially, attempting to deliver the current payload in the infinite loop, which causing more OpenSearch overload.
The log records have also influence to the overload in case, if the target OpenSearch cluster is using for storing logs as well.

To Reproduce
Steps to reproduce the behavior:
Lower OpenSearch cluster to best possible minimum and start sending huge amount of data from DataPrepper (OTLP metrics and traces) and also have some log processor active to feeding the same OpenSearch cluster (Logstash, FluentD, ...).

Expected behavior
When DataPrepper got reject from OpenSearch, it should waiting some time, before attempting to retrying delivery.

Screenshots
Image

Environment (please complete the following information):

  • Kubernetes: 1.34
  • Version DataPrepper (official Docker image): 2.7.0
  • Version OpenSearch (official Docker image): 2.16

Additional context
As you can see from screenshot, the timestamp of the log records are basically same, so DataPrepper not attempting to wait and only crashing to overloaded OpenSearch cluster.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with a minimally sized OpenSearch cluster while sending large OTLP metrics and traces and feeding logs to the same cluster. Trace DataPrepper's OpenSearch delivery path and its handling of rejected payloads. Done means rejected deliveries produce a warning and wait with exponential backoff before retrying, without continuously worsening the overload.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
backend, data-engineering, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.