aws / aws/containers-roadmap

Unusual/Complex ECS CloudFormation rollback behavior

Open
#117 3 comments 4 reactions 0 assignees View on GitHub
Deployments ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

The rollback procedure that ECS uses when a CloudFormation update is cancelled seems confusing and unnecessarily complex to me.

Here is a scenario where I have a CloudFormation update that I need to cancel because the new task definition has a bug and keeps failing to start:

**Expected behavior**
0) I manually cancel the CF template update and the rollback begins
1) ECS stops attempting to deploy the new (bad) task definition
2) All other CloudFormation resources get rolled back
3) My original (good) task definition is still running fine, so there was no downtime

**Actual behavior**
0) I manually cancel the CF template update and the rollback begins
1) ECS stops attempting to deploy the new (bad) task definition
2) ECS deploys a second copy of the old (good) task definition (WHY!?!?)
3) ECS stops the first copy of my old (good) task definition (WHY!?!?)
4) All other CloudFormation resources get rolled back

This might seem innocuous enough, however because of how I manage my container instances it creates a considerable amount of churn. Here is what actually happens

**Initial state**: ECS Task Definition 1 is currently running on Container Instance 1

1) I update the CF template which triggers the creation of Task Definition 2 and a new AutoScaling Group
2) The new ASG launches Container Instance 2 so that the new ECS Service/Task has somewhere to be deployed
3) ECS attempts to deploy Task Definition 2 on Container Instance 2 but the new task has a bug so it keeps failing
4) I manually cancel the CF template update and the rollback begins
5) ECS stops attempting to deploy Task Definition 2
6) ECS deploys a second copy of Task Definition 1 on Container Instance 2 (WHY!?!?)
7) ECS stops Task Definition 1 on Container Instance 1 (WHY!?!?)
8) CloudFormation starts rolling back all other resources so it attempts to delete the new ASG and Container Instance 2
9) Thankfully, I have a lifecycle hook which pauses termination and marks Container Instance 2 as DRAINING
10) ECS sees that Task Definition 1 is on a DRAINING instance so it starts Task Definition 1 on Container Instance 1 (again) and stops the copy that is running on Container Instance 2
11) The lifecycle hook completes once all tasks are stopped on Container Instance 2 and the instance is shut down
12) All other CloudFormation resources get rolled back

Thanks to my lifecycle hook, no downtime actually happens, but it still seems like an unnecessarily complex sequence, and I had to jump through a lot of hoops to get the lifecycle hook working in this scenario.

Is this the expected behavior? Is there any way to trigger the simpler approach?

Contributor guide

Open the contributing guide

Research direction

No repository files or tests are named. Start by reviewing the ECS service deployment and CloudFormation rollback behavior, then reproduce the cancelled-update scenario with the Auto Scaling lifecycle hook and task-definition sequence described here. Done means determining whether the duplicate deployment is expected and documenting or identifying a supported simpler rollback approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.