awslabs / awslabs/aws-solutions-constructs

New feature: aws-stepfunctionstasks-sqs

Open
#280 2 comments 0 reactions 0 assignees View on GitHub
feature-request needs-triage
Dominant language
TypeScript
Stars
1.4k
Forks
268
Avg merge
5h 18m
Merged PRs (30d)
5

Description

Step function has a callback pattern called 'Callback with Task Token' - https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html. It allows for slower back-ends to respond rather than getting throttled and fail.

###Use Case
Most back-ends and AWS services have TPS limits may not be able to handle bursts or higher volume of streaming data. Having step function workflow, retries and data streams like MSK or Kinesis does reduce the throttling issues. But does not solve the problem where a step function task has a slower back-end and a successful completion of a task requires transitioning through all the states in the statemachine.

###Proposed Solution
To use the callback pattern, stepfunctions can have task that is backed with a queue (like SQS). On the consumer side of the queue, the target resource (Ex: lambda, a containerized implementation, or an EC2 instance, processes the message and sends the status back to the state. This triggers the state machine to transition to the next step.

The aws-stepfunctions-sqs (a statemachine task that is backed by SQS) can be combined with the existing [aws-sqs-lambda](https://github.com/awslabs/aws-solutions-constructs/tree/main/source/patterns/%40aws-solutions-constructs/aws-sqs-lambda) construct to complete this pattern (lambda specific version of the stepfunction callback pattern). Note: additional permissions are required to be assigned, in this case to the lambda function to be able to send back the task status to the statemachine.

As part of this construct, the proposal is to create a state machine fragment (https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-stepfunctions.StateMachineFragment.html) that would allow it to be chained to a state machine definition.

###Other
https://zaccharles.medium.com/async-callbacks-with-aws-step-functions-and-task-tokens-9df97fe8973c
https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html
https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-stepfunctions.StateMachineFragment.html

###Readme
here are more details about the actual construct with its [`README`](https://github.com/knihit/aws-solutions-constructs/tree/master/source/patterns/%40aws-solutions-constructs/aws-stepfunctionstask-sqs) and architecture diagram.

* [x] :wave: I may be able to implement this feature request
* [ ] :warning: This feature might incur a breaking change

---

This is a :rocket: Feature Request

Contributor guide

Open the contributing guide

Research direction

Start with the proposed aws-stepfunctionstask-sqs README and architecture diagram, then compare the existing aws-sqs-lambda construct and the AWS StateMachineFragment and callback-task-token documentation. Confirm the construct's queue, consumer integration, task-status permissions, and state-machine chaining requirements; done means the proposed pattern is implemented and documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.