opensearch-project / opensearch-project/data-prepper

[BUG] The BufferAccumulator class can infinitely backoff and retry and will not throw exception on shutdown

Open
#6,272 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
The BufferAccumulator class does backoff and retry to write and flush to the buffer. When the buffer is full or circuit breaker is open, the buffer will reject the writes and the buffer accumulator will backoff and retry infinitely.

However, when there is a shutdown command to Data Prepper, the BufferAccumulator will not stop backing off and retrying, which will lead to the threads using the BufferAccumulator to block and not exit, delaying the shutdown of the source (potentially for a long time if the buffer never empties or circuit breaker never recovers)

When circuit breaker opens, for example, this exception will get thrown (https://github.com/opensearch-project/data-prepper/blob/ca1552e8bce52f0e8b1b3ffe84557fd890a8449e/data-prepper-core/src/main/java/org/opensearch/dataprepper/core/parser/CircuitBreakingBuffer.java#L62) to the BufferAccumulator and will be caught here (https://github.com/opensearch-project/data-prepper/blob/ca1552e8bce52f0e8b1b3ffe84557fd890a8449e/data-prepper-plugins/buffer-common/src/main/java/org/opensearch/dataprepper/buffer/common/BufferAccumulator.java#L90) and then will retry infinitely

To Reproduce
Steps to reproduce the behavior:

  1. Configure a low circuit breaker threshold in the data-prepper-config.yaml
  2. Start a pipeline that uses buffer accumulator
  3. Make sure circuit breaker is open.
  4. Call shutdown and see that Data Prepper does not exit until the source forces shutdown on the threads (if the source does so)

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • OS: [e.g. Ubuntu 20.04 LTS]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in data-prepper-plugins/buffer-common/src/main/java/org/opensearch/dataprepper/buffer/common/BufferAccumulator.java around line 90, then inspect data-prepper-core/src/main/java/org/opensearch/dataprepper/core/parser/CircuitBreakingBuffer.java around line 62. Trace how shutdown reaches the retry loop and determine the expected shutdown signal. Done means shutdown no longer leaves BufferAccumulator callers blocked indefinitely when the buffer rejects writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.