EKS BUG: Metrics server is not connecting to new node automatically after updating/downgrading nodegroup version
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
Metrics server is still trying to connect previous node after updating/downgrading nodegroup version but when restarting the metrics server deployment it is working fine. So just want to know why there is need to restart deployment.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Created EKS nodegroup running on version "1.27.4-20230825" and installed the metrics server from below doc :
https://docs.aws.amazon.com/eks/latest/userguide/metrics-server.html
Here metrics server is working fine and no error in logs of pod. Then I have updated the nodegroup version to "1.27.5-20231002" and observed that metrics pods are running on new nodes but I have observed the error in metrics server pod log :
$ kubectl logs metrics-server-5d875656f5-5xj8m -n kube-system
I1020 12:06:00.068272 1 serving.go:342] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I1020 12:06:00.645219 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1020 12:06:00.645243 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I1020 12:06:00.645408 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1020 12:06:00.645496 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1020 12:06:00.645570 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1020 12:06:00.645589 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1020 12:06:00.645951 1 secure_serving.go:267] Serving securely on [::]:4443
I1020 12:06:00.647315 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
W1020 12:06:00.656508 1 shared_informer.go:372] The sharedIndexInformer has started, run more than once is not allowed
I1020 12:06:00.648121 1 dynamic_serving_content.go:131] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
I1020 12:06:00.746099 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I1020 12:06:00.746098 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1020 12:06:00.746116 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
E1020 12:07:00.644194 1 scraper.go:140] "Failed to scrape node" err="Get \"https://172.31.14.189:10250/metrics/resource\": dial tcp 172.31.14.189:10250: connect: connection refused" node="ip-172-31-14-189.eu-central-1.compute.internal"
E1020 12:07:29.140632 1 scraper.go:140] "Failed to scrape node" err="Get \"https://172.31.14.189:10250/metrics/resource\": context deadline exceeded" node="ip-172-31-14-189.eu-central-1.compute.internal"
E1020 12:07:44.140409 1 scraper.go:140] "Failed to scrape node" err="Get \"https://172.31.14.189:10250/metrics/resource\": context deadline exceeded" node="ip-172-31-14-189.eu-central-1.compute.internal"
E1020 12:07:59.138903 1 scraper.go:140] "Failed to scrape node" err="Get \"https://172.31.14.189:10250/metrics/resource\": context deadline exceeded" node="ip-172-31-14-189.eu-central-1.compute.internal"
**Are you currently working around this issue?**
To resolve this error we need to restart the metrics server deployment. So my query is why there is need to restart deployment it should fetch data from new nodes.
**Additional context**
Anything else we should know?
**Attachments**
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)
Contributor guide
Research direction
Start with the AWS EKS metrics-server documentation and the provided kubectl logs, focusing on scraper.go:140 and the node endpoint at port 10250. Reproduce an EKS nodegroup version update or downgrade and check whether metrics-server refreshes node targets without a deployment restart. Done means the cause is identified and the issue is either fixed or clearly documented with the required behavior and workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- infrastructure, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100