opensearch-project / opensearch-project/data-prepper

[BUG]DynamoDB source: Stream shards never consumed after export completion when export takes >24h

Open
#7,150 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug untriaged
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
When a DynamoDB source pipeline has both export and stream configured, and the export takes longer than 24 hours to complete, stream processing never starts even after the export fully completes. The pipeline logs "All Exports are done, streaming can continue..." but stream shards are never acquired.

To Reproduce
Steps to reproduce the behavior:
Create OSIS pipeline with DynamoDB source having both export and stream configured
Point to a large DynamoDB table (>10M records) where export takes >24h
Start the pipeline
Wait for export to complete (monitor "All Exports are done" log)
Observe "No new shards acquired after N attempts" continues indefinitely after export completion
Confirm aws_lambda_processor.recordsIn.count metric stays at 0 for stream records
Confirm DynamoDB list-exports shows all exports as COMPLETED

Expected behavior

The weird behavior with the pipeline logs that I am seeing is:

Despite export being completed 15 mins ago

2026-09-18T00:19:39.417 [acknowledgement-callback-6] INFO  org.opensearch.dataprepper.plugins.source.dynamodb.export.DataFileScheduler - All Exports are done, streaming can continue. 
2026-09-18T00:31:41.544 [pool-24-thread-4] INFO  org.opensearch.dataprepper.plugins.source.dynamodb.stream.StreamScheduler - No new shards acquired after 50 attempts. This means that all shards are currently being consumed, or that the export is still in progress. New shards will not be consumed until the export is fully processed.

Screenshots

Image

If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • OSIS managed pipeline
  • DynamoDB source with both export and stream configured
  • Table: ~10M+ records (export takes 48+ hours)
Relevant PRs: #6892, #6651, #7070

https://github.com/opensearch-project/data-prepper/issues/6892
https://github.com/opensearch-project/data-prepper/issues/6651
https://github.com/opensearch-project/data-prepper/issues/7070

Questions:

  1. Can we not acquire shard partitions until export is done
    Move the export gate from ShardConsumer.waitForExport() to StreamScheduler.run(). Don't acquire partitions at all during export - no leases held, no shards expiring while owned, no broken shard chains.

// Only then acquire and process partitions

  1. Did any of the recent PRs update isExportDone() reliability?
    Current behavior: isExportDone() return false forever despite export completion

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with DataFileScheduler and StreamScheduler, especially StreamScheduler.run(), ShardConsumer.waitForExport(), and isExportDone(). Review the behavior described in related PRs #6892, #6651, and #7070, then reproduce the >24-hour export scenario. Done means stream shards are acquired after export completion and aws_lambda_processor.recordsIn.count increases for stream records.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, data, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.