opensearch-project / opensearch-project/data-prepper

[BUG] Ownership can timeout on full buffer for pull based sources

Open
#4,422 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
we currently update source coordination ownership for partitions synchronously in pull based sources like S3, OpenSearch, and DynamoDB. This happens in a loop approximately every 2 minutes, but when the buffer is very full, we spend time retrying to write to the buffer, which leads to expiring ownership of the partition, and reprocessing of that partition by another node of Data Prepper

Expected behavior
Asynchronously update ownership every 2 minutes without depending on the primary loop. For example, this is done here for DynamoDB (https://github.com/opensearch-project/data-prepper/blob/a20756cc13e6f7b7f088544df03bb1230a88af8f/data-prepper-plugins/dynamodb-source/src/main/java/org/opensearch/dataprepper/plugins/source/dynamodb/export/DataFileLoader.java#L206). We should update ownership in a timely manner regardless of how long it takes to write to the buffer.

Alternative consideration
Increase the ownership timeout to be a higher value or check ownership updates in between attempts to write to the buffer

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • OS: [e.g. Ubuntu 20.04 LTS]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the DynamoDB example in dynamodb-source/src/main/java/org/opensearch/dataprepper/plugins/source/dynamodb/export/DataFileLoader.java around line 206, then inspect ownership updates and buffer-writing loops for the pull-based sources named in the issue. Done means ownership is refreshed asynchronously on schedule without depending on buffer write duration, preventing partition expiry and reprocessing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.