aws / aws/eks-anywhere

Cluster-autoscaler errors on permissions

Open
#6,064 3 comments 0 reactions 0 assignees View on GitHub
external
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:

I follow the docs and the ADC for getting cluster autoscaler to function, but it appears to be missing some permissions, I've checked the curated patches and I don't see how it could be working.

```
3
I0621 11:28:57.621036 1 clusterapi_provider.go:67] discovered node group: MachineDeployment/eksa-system/w01-md-0 (min: 1, max: 30, replicas: 3)
22
I0621 11:28:57.625886 1 clusterapi_provider.go:67] discovered node group: MachineDeployment/eksa-system/w01-md-0 (min: 1, max: 30, replicas: 3)
21
I0621 11:28:57.625931 1 clusterapi_controller.go:726] waiting for cache sync on infrastructure resource
20
I0621 11:28:57.626039 1 reflector.go:205] Reflector from k8s.io/client-go/dynamic/dynamicinformer/informer.go:91 configured with expectedType of *unstructured.Unstructured with empty GroupVersionKind.
19
I0621 11:28:57.626068 1 reflector.go:221] Starting reflector *unstructured.Unstructured (0s) from k8s.io/client-go/dynamic/dynamicinformer/informer.go:91
18
I0621 11:28:57.626075 1 reflector.go:257] Listing and watching *unstructured.Unstructured from k8s.io/client-go/dynamic/dynamicinformer/informer.go:91
17
W0621 11:28:57.627058 1 reflector.go:424] k8s.io/client-go/dynamic/dynamicinformer/informer.go:91: failed to list *unstructured.Unstructured: vspheremachinetemplates.infrastructure.cluster.x-k8s.io is forbidden: User "system:serviceaccount:kube-system:cluster-autoscaler" cannot list resource "vspheremachinetemplates" in API group "infrastructure.cluster.x-k8s.io" at the cluster scope
16
E0621 11:28:57.627094 1 reflector.go:140] k8s.io/client-go/dynamic/dynamicinformer/informer.go:91: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: vspheremachinetemplates.infrastructure.cluster.x-k8s.io is forbidden: User "system:serviceaccount:kube-system:cluster-autoscaler" cannot list resource "vspheremachinetemplates" in API group "infrastructure.cluster.x-k8s.io" at the cluster scope
15
I0621 11:28:58.984700 1 reflector.go:257] Listing and watching *unstructured.Unstructured from k8s.io/client-go/dynamic/dynamicinformer/informer.go:91
```

**What you expected to happen**:

I'd expect the error messages to not exist and scaling to function.

**How to reproduce it (as minimally and precisely as possible)**:

Deploy a cluster with the annotations and install cluster autoscaler.

**Anything else we need to know?**:

It looks related to https://github.com/kubernetes/autoscaler/issues/5509

Ideally I'd just like you to confirm if this works in your labs/tests.

**Environment**:
- EKS Anywhere Release: 0.16.0
- EKS Distro Release: 1.27

Contributor guide

Open the contributing guide

Research direction

Start with the cluster-autoscaler logs, especially clusterapi_provider.go:67 and clusterapi_controller.go:726, and compare the documented ADC setup with the curated patches. Reproduce the forbidden list/watch request for vspheremachinetemplates and verify whether the cluster-autoscaler service account permissions cover it. Done means the permission errors no longer occur and scaling functions as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.