awslabs / awslabs/aws-solutions-constructs
New feature: aws-stepfunctionstasks-sqs
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 268
- Avg merge
- 5h 18m
- Merged PRs (30d)
- 5
Description
Step function has a callback pattern called 'Callback with Task Token' - https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html. It allows for slower back-ends to respond rather than getting throttled and fail.
###Use Case
Most back-ends and AWS services have TPS limits may not be able to handle bursts or higher volume of streaming data. Having step function workflow, retries and data streams like MSK or Kinesis does reduce the throttling issues. But does not solve the problem where a step function task has a slower back-end and a successful completion of a task requires transitioning through all the states in the statemachine.
###Proposed Solution
To use the callback pattern, stepfunctions can have task that is backed with a queue (like SQS). On the consumer side of the queue, the target resource (Ex: lambda, a containerized implementation, or an EC2 instance, processes the message and sends the status back to the state. This triggers the state machine to transition to the next step.
The aws-stepfunctions-sqs (a statemachine task that is backed by SQS) can be combined with the existing [aws-sqs-lambda](https://github.com/awslabs/aws-solutions-constructs/tree/main/source/patterns/%40aws-solutions-constructs/aws-sqs-lambda) construct to complete this pattern (lambda specific version of the stepfunction callback pattern). Note: additional permissions are required to be assigned, in this case to the lambda function to be able to send back the task status to the statemachine.
As part of this construct, the proposal is to create a state machine fragment (https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-stepfunctions.StateMachineFragment.html) that would allow it to be chained to a state machine definition.
###Other
https://zaccharles.medium.com/async-callbacks-with-aws-step-functions-and-task-tokens-9df97fe8973c
https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html
https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-stepfunctions.StateMachineFragment.html
###Readme
here are more details about the actual construct with its [`README`](https://github.com/knihit/aws-solutions-constructs/tree/master/source/patterns/%40aws-solutions-constructs/aws-stepfunctionstask-sqs) and architecture diagram.
* [x] :wave: I may be able to implement this feature request
* [ ] :warning: This feature might incur a breaking change
---
This is a :rocket: Feature Request
Contributor guide
Research direction
Start with the proposed aws-stepfunctionstask-sqs README and architecture diagram, then compare the existing aws-sqs-lambda construct and the AWS StateMachineFragment and callback-task-token documentation. Confirm the construct's queue, consumer integration, task-status permissions, and state-machine chaining requirements; done means the proposed pattern is implemented and documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100