opensearch-project / opensearch-project/data-prepper

[BUG] S3-SQS source pipelines can get stuck when SQS is reading from multiple buckets and one bucket has permissions issues

Open
#3,930 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
S3-SQS source pipelines can get stuck when SQS is reading from multiple buckets and one bucket has permissions issues.

To Reproduce
Steps to reproduce the behavior:

  1. Create an SQS queue
  2. Create 2 S3 buckets and configure event notifications for the SQS queue in step 1
  3. Create an IAM role with permissions needed to get objects from one of the S3 buckets (call it Bucket1)
  4. Create an S3-SQS pipeline using the SQS queue and IAM role
  5. Upload objects to Bucket1, the pipeline should process them correctly
  6. Upload objects to Bucket 2, the pipeline will get 403 errors and retry
  7. Upload objects to Bucket1, these will not be processed because the pipeline is still trying to process objects from Step 6

Expected behavior
If a pipeline hits access denied errors with one bucket it should still process objects from the other bucket

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files or tests; begin by locating the S3-SQS source pipeline implementation and reproducing the two-bucket IAM-permission scenario. Done means objects from the permitted bucket continue processing while access-denied messages from the other bucket do not block the pipeline.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.