opensearch-project / opensearch-project/data-prepper
[BUG]DynamoDB source: Stream shards never consumed after export completion when export takes >24h
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
When a DynamoDB source pipeline has both export and stream configured, and the export takes longer than 24 hours to complete, stream processing never starts even after the export fully completes. The pipeline logs "All Exports are done, streaming can continue..." but stream shards are never acquired.
To Reproduce
Steps to reproduce the behavior:
Create OSIS pipeline with DynamoDB source having both export and stream configured
Point to a large DynamoDB table (>10M records) where export takes >24h
Start the pipeline
Wait for export to complete (monitor "All Exports are done" log)
Observe "No new shards acquired after N attempts" continues indefinitely after export completion
Confirm aws_lambda_processor.recordsIn.count metric stays at 0 for stream records
Confirm DynamoDB list-exports shows all exports as COMPLETED
Expected behavior
The weird behavior with the pipeline logs that I am seeing is:
Despite export being completed 15 mins ago
2026-09-18T00:19:39.417 [acknowledgement-callback-6] INFO org.opensearch.dataprepper.plugins.source.dynamodb.export.DataFileScheduler - All Exports are done, streaming can continue.
2026-09-18T00:31:41.544 [pool-24-thread-4] INFO org.opensearch.dataprepper.plugins.source.dynamodb.stream.StreamScheduler - No new shards acquired after 50 attempts. This means that all shards are currently being consumed, or that the export is still in progress. New shards will not be consumed until the export is fully processed.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
- OSIS managed pipeline
- DynamoDB source with both export and stream configured
- Table: ~10M+ records (export takes 48+ hours)
Relevant PRs: #6892, #6651, #7070
https://github.com/opensearch-project/data-prepper/issues/6892
https://github.com/opensearch-project/data-prepper/issues/6651
https://github.com/opensearch-project/data-prepper/issues/7070
Questions:
- Can we not acquire shard partitions until export is done
Move the export gate from ShardConsumer.waitForExport() to StreamScheduler.run(). Don't acquire partitions at all during export - no leases held, no shards expiring while owned, no broken shard chains.
// Only then acquire and process partitions
- Did any of the recent PRs update isExportDone() reliability?
Current behavior:isExportDone()return false forever despite export completion
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with DataFileScheduler and StreamScheduler, especially StreamScheduler.run(), ShardConsumer.waitForExport(), and isExportDone(). Review the behavior described in related PRs #6892, #6651, and #7070, then reproduce the >24-hour export scenario. Done means stream shards are acquired after export completion and aws_lambda_processor.recordsIn.count increases for stream records.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, data, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100