[ECS] [request]: Prevent excessive interruption during ASG rolling update
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
During an ASG update using a rolling update policy, or instance refresh, prevent tasks from being placed on instances that will be replaced as part of the update.
**Which service(s) is this request for?**
ECS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
When updating ASGs using CloudFormation, we can define an UpdatePolicy. This UpdatePolicy can make it so that during the CFN update of the ASG, all the existing instances are replaced with new ones that use the new AMI/launch template/etc. This can be configured to be done in a rolling fashion so that batches of instances are replaced at a time.
When combined with ECS' managed draining and termination protection, this makes it quite easy to roll out changes to ASGs used by ECS capacity providers. However, during a rolling update, ECS won't know what instances _will be_ terminated in a future batch. If we consider a rolling update policy where we start out by terminating 10 out of 100 old instances in the ASG, the ASG will launch 10 new instances, making for a total of 120 instances of which 110 are not in termination:pending state. This means that any task that is running on the 10 instances being terminated now has a 91% chance to be placed on an instance that will be terminated in a future batch and a 9% chance to end up on one of the new instances. Given a placement strategy such as binpack, it's in fact far more likely that they will be placed on older instances that will be terminated.
Overall this means that tasks will experience a lot of interruption as they won't just be started and stopped once, but potentially X times where X is the number of batches that the ASG update is done in.
**Are you currently working around this issue?**
Today we are not using an ASG UpdatePolicy because of this issue. Instead, we have to build our own workflow that lists all the instances in the ASG, add a "tainted" attribute to them that is used in all service placement constraints to prevent new tasks from being launched on them, split the list of instances into batches and terminate one batch at a time. That way we will be moving all the tasks from one batch of old instances into one batch of new instances, instead of moving them from old instances to other old instances over and over again.
**Additional information**
While I'm referring to the ASG UpdatePolicy here, the same thing applies to instance refresh.
Contributor guide
Assessment
This issue has not been assessed yet.