[EKS] [request]: Notifications / More control over EKS Control Plane Node Patch Rollouts
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
It would be great to be notified in advance, and ideally be able to approve/schedule, before any patches are applied to the EKS master nodes.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
We had a number of CoreDNS restarts this morning which were caused by a bug still present in the AWS recommended version of CoreDNS (1.3.1) for Kubernetes 1.14. Loss of DNS affected a number of our applications. From the cloudwatch EKS logs, we traced this down to a rollout / restart of the EKS master nodes. This example is about CoreDNS, but it's not unlikely that other components that are installed on a cluster could suffer similar effects.
Basically it would be nice to ensure that any master patch rollouts were controllable so that we can test on our dev / stg environments before they go to prd. At the moment it's a lottery as to which of our environments get patched first.
Worst case it would be beneficial to be notified when a particular cluster's master nodes will be patched in advance, so we can at least keep an eye on it or quickly diagnose any issues that occur as a result.
**Are you currently working around this issue?**
The only thing we can think of doing currently is creating a custom metric from the EKS logs in cloudwatch to alert us when master nodes restart, or implementing some additional blackbox testing. This would at least tell us that the event has occurred without having to go through the logs, though wouldn't give us any advance knowledge that it was going to occur.
**Additional context**
CoreDNS issue:
https://github.com/coredns/coredns/issues/2629
AWS Docs containing version recommendations for k8s addons: https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html
Contributor guide
Assessment
This issue has not been assessed yet.