[ECS] [request]: Abandon termination if draining fails for managed hosts
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
We would like to use managed instance draining, but if draining fails we do not want to terminate the healthy instances.
From https://docs.aws.amazon.com/AmazonECS/latest/developerguide/managed-instance-draining.html:
> The container instance remains in the DRAINING state until either all tasks are stopped or 48 hours has passed. Daemon tasks are the last to stop after all replica tasks have stopped.
If we are stuck in the `DRAINING` state for 48 hours, that is a signal of an unhealthy state, and we would like to abort the draining.
**Which service(s) is this request for?**
ECS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
We deploy a critical service as a `DAEMON` process, and our services cannot start on EC2 hosts if this `DAEMON` isn't running. We also want to do automated rolling updates of our EC2 instances. However, if an unattended upgrade creates new EC2 hosts with a failed `DAEMON`, we do not want ECS to terminate the healthy hosts, and instead remain in the `DRAINING` state, or automatically abort `DRAINING` the instance.
**Are you currently working around this issue?**
Using a custom termination policy on the ASG that does not return any instances if the cluster isn't healthy: https://docs.aws.amazon.com/autoscaling/ec2/userguide/lambda-custom-termination-policy.html, and lambdas that implement custom draining logic. We are not using managed EC2 instances with ECS.
**Additional context**
This is related to https://github.com/aws/containers-roadmap/issues/2467, but it is the inverse. Being able to configure "no timeout" would solve this problem.
Contributor guide
Research direction
Start with the AWS ECS managed instance draining documentation linked in the issue, then compare the related roadmap issue #2467. Define how a failed or 48-hour drain should prevent termination, including whether remaining in DRAINING or configuring no timeout is the intended outcome; the issue names no repository files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100