opensearch-project / opensearch-project/data-prepper
[BUG] S3-SQS source pipelines can get stuck when SQS is reading from multiple buckets and one bucket has permissions issues
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
S3-SQS source pipelines can get stuck when SQS is reading from multiple buckets and one bucket has permissions issues.
To Reproduce
Steps to reproduce the behavior:
- Create an SQS queue
- Create 2 S3 buckets and configure event notifications for the SQS queue in step 1
- Create an IAM role with permissions needed to get objects from one of the S3 buckets (call it Bucket1)
- Create an S3-SQS pipeline using the SQS queue and IAM role
- Upload objects to Bucket1, the pipeline should process them correctly
- Upload objects to Bucket 2, the pipeline will get 403 errors and retry
- Upload objects to Bucket1, these will not be processed because the pipeline is still trying to process objects from Step 6
Expected behavior
If a pipeline hits access denied errors with one bucket it should still process objects from the other bucket
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files or tests; begin by locating the S3-SQS source pipeline implementation and reproducing the two-bucket IAM-permission scenario. Done means objects from the permitted bucket continue processing while access-denied messages from the other bucket do not block the pipeline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100