aws / aws/containers-roadmap

[EKS] [request]: More visibility into why EKS control plane nodes are shutting down unexpectedly

Open
#1,878 0 comments 9 reactions 0 assignees View on GitHub
EKS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
A dashboard, or access to some logs, or a notification before updating EKS components that might cause outages.

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Since yesterday around 7pm Pacific we are seeing EKS control plane outages (one time event per cluster, 3 clusters so far), the controllers are unable to connect to the API server, and they are failing to renew their leases. The outage last minutes. It would be great if we could have more visibility into why this is happening, or notifications being sent out about the changes being made to the control plane that might cause outages.

**Are you currently working around this issue?**
There's nothing we are able to do at the moment to resolve this issue, other than waiting it out.

**Additional context**
Logs suggest that there was a shutdown at that time.

**Attachments**
Sample logs from Controller manager

```
E1101 17:34:16.492796 12 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Get "https://172.16.57.106:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 172.16.57.106:443: connect: connection refused

E1101 17:34:12.937494 10 horizontal.go:227] unable to determine resource for scale target reference: Get "https://172.16.57.106:443/api?timeout=32s": dial tcp 172.16.57.106:443: connect: connection refused
```

Contributor guide

Open the contributing guide

Research direction

No repository files, tests, or entry points are identified; begin by reviewing the EKS control-plane shutdown behavior and the supplied controller-manager logs. Define whether the request is for dashboards, logs, or notifications, and treat documented visibility into the cause and relevant updates as the completion criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.