cdklabs / cdklabs/construct-hub
Refactor doc generation processes to eliminate exponential backoff in step functions
- Dominant language
- TypeScript
- Stars
- 239
- Forks
- 30
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 11
Description
## Background
When the doc generation process has issues, it's possible for its DLQ to get very large. (For example, in the Gamma account the DLQ currently has 20,000+ messages). When a Lambda is invoked to redrive these messages, it simply initializes thousands of step function executions, which delegate ECS tasks for generating documentation for individual languages. Since this spike of workload is far greater than what ECS can handle at once, the step function frequently gets throttled, so it has been configured with a very slow but exponential backoff:
https://github.com/cdklabs/construct-hub/blob/02d5ff3a01768b9203ac061be7ed90200f714e92/src/backend/orchestration/index.ts#L40
## Problem
This backoff-and-retry solution causes the history of individual executions gets cluttered with dozens of failed ECS task requests, and moreover, the larger the number of concurrent tasks that are being executed, the longer it will take for the state machine executions to reach a point in time where individual tasks make progress.
## Proposed solution
We can make this architecture more stable by introducing a queue, and a lambda function that polls from the queue and starts tasks whenever we are not close to hitting the task limit (e.g. below 80% capacity). In the state machine that spawns the individual tasks, instead of invoking the ECS task directly, it will add the tasks to an SQS queue, and then wait to receive a response before continuing the state machine, in accordance with the pattern described here: https://docs.aws.amazon.com/step-functions/latest/dg/callback-task-sample-sqs.html We can assume if the task execution does not hear a response for at least 6 hours, then it has failed and should go into the DLQ.
This would eliminate all errors caused by trying to invoke too many ECS tasks at once, while also giving us greater flexibility over how much compute we should allocate to the ECS cluster (we could likely set up autoscaling based on the amount of items in the queue).
## Open Questions
The proposed solution is a somewhat common serverless architecture pattern - is there an existing CDK L3 construct we can use to abstract this behavior? If not, could we create one?
Contributor guide
Research direction
Start with src/backend/orchestration/index.ts around the linked exponential-backoff configuration, then read the AWS Step Functions callback-task sample referenced in the issue. Done means the orchestration no longer relies on repeated ECS invocation failures and instead supports queued work, capacity-aware task starts, callback responses, and a six-hour failure path to the DLQ.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- backend, cloud
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100