awslabs / awslabs/aws-solutions-constructs
S3ToStepFunctions: add a SQS FIFO Queue to get exactly-once delivery
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 268
- Avg merge
- 5h 18m
- Merged PRs (30d)
- 5
Description
In using this construct I found I was spending a lot more than expected because I was getting many duplicate events, and running my ML step function is quite expensive. I suggest adding a FIFO queue to get exactly-once delivery semantics (or change to an S3-SNS-SQS-Stepfunction workflow).
This is a :rocket: Feature Request
Contributor guide
Research direction
Start at the S3ToStepFunctions construct and inspect its current event-delivery path. Compare the proposed SQS FIFO option with the alternative S3-SNS-SQS-StepFunctions workflow, then define which approach provides the requested delivery semantics and what behavior should demonstrate completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100