aws / aws/aws-node-termination-handler

Add custom delay for instance refresh actions

Open
#465 5 comments 0 reactions 0 assignees View on GitHub
Priority: Medium stalebot-ignore Type: Enhancement
Dominant language
Go
Stars
1.8k
Forks
286
PR merge metrics
No merged PRs in 30d

Description

When using instance refresh to update ASGs it looks like the events come through with a start date of now which triggers the node-termination handler to start cordoning and draining the node immediately. This does work correctly if the ASG healthy percentage is set to 100% and all pods have replicas and PDBs (for NTH we need https://github.com/aws/aws-node-termination-handler/pull/463 to satisfy this); but single pods such as Prometheus will often be un-schedulable for a short period while the new node boots up.

To make this whole process function without any downtime a custom duration to wait on ASG termination events could be adopted and defaulted to something like 90 seconds. Assuming that this wait time was longer than the time to start and join a node to the cluster there would be no un-schedulable pods and the ability to use a non 100% ASG healthy percentage. Combined with the ASG lifecycle hook timeout this would support a high level of customisation without much extra complexity.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.