[ECS] [request]: Configurable timeout for managed draining lifecycle hook
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
Allow configuring the timeout of the EC2 lifecycle hook after which the termination continues even if there are still running tasks on the instance.
**Which service(s) is this request for?**
ECS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
We manage large multi-tenant cluster and use the managed draining functionality to ensure that tasks running on instances are gracefully terminated before allowing instance termination. What sometimes ends up happening is that some service running on the cluster fails to start a replacement task on another instance, which leaves the instance unable to be terminated for 48 hours until the global EC2 lifecycle hook timeout when the termination proceeds anyway. Given the number of services and clusters we operate, this makes rolling out new AMIs or other platform changes very challenging as all it takes is a single service to be in a bad state where it can't be replaced to hold up the ASG instance replacement for 48 hours.
**Are you currently working around this issue?**
Before using managed instance draining, we had our own custom lifecycle hook doing the same thing, but there we configured a timeout where we continued the termination after X minutes even if we hadn't managed to drain all tasks from the instance, accepting that those services might incur downtime if they don't make sure their service is able to start up on a new instance at any time.
The workaround is to provision our own custom lifecycle hook that does 99% of the same work that the managed one does to check if there are still running tasks on the instance, except that it also signals `ABANDON` after a defined timeout period, to continue the instance termination even if other lifecycle hooks are still pending.
Contributor guide
Assessment
This issue has not been assessed yet.