opensearch-project / opensearch-project/data-prepper
[BUG] DynamoDB source should not complete all partitions when receiving errors from GetShardIterator
@graytaylor0 is already working on this.
Since Nov 4, 2025.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
The ShardConsumerFactory class returns null if it receives any exceptions on the GetShardIterator call, and then the StreamScheduler completes the shard partition when that happens. We should not complete the partition for the shard when we receive exceptions like InternalServerError or AccessDenied, to name a few
Expected behavior
Exceptions like InternalServerError or AccessDenied should not result in the partition being completed, and should eventually be retried.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
- OS: [e.g. Ubuntu 20.04 LTS]
- Version [e.g. 22]
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.