kubernetes / kubernetes/kops

Cluster validation cannot complete if metrics-server addon is enabled and there are less than 2 non-master nodes

Open
#16,585 17 comments 0 reactions 0 assignees View on GitHub
kind/bug lifecycle/rotten
Dominant language
Go
Stars
16.7k
Forks
4.7k
Avg merge
17h 21m
Merged PRs (30d)
68

Description

/kind bug

**1. What `kops` version are you running? The command `kops version`, will display
this information.**
1.28.4

**2. What Kubernetes version are you running? `kubectl version` will print the
version if a cluster is running or provide the Kubernetes version specified as
a `kops` flag.**
1.28.10

**3. Relevant cluster manifest portion:**
```yaml
metricsServer:
enabled: true
insecure: false
```

**9. Anything else do we need to know?**
If the metrics server addon is enabled and there are less than two non-master nodes, then cluster validation will be failing indefinitely:

```
I0521 18:26:05.985979 76254 instancegroups.go:563] Cluster did not pass validation, will retry in "30s": system-cluster-critical pod "metrics-server-5c45c474f5-t8ppf" is pending.
```

The reason for this is that the deployment manifest specifies that there must be two replicas and topology spread constraints are defined in such a way that these two replicas must run on different nodes:

```yaml
spec:
...
replicas: 2
...
template:
...
spec:
...
topologySpreadConstraints:
- labelSelector:
matchLabels:
k8s-app: metrics-server
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
- labelSelector:
matchLabels:
k8s-app: metrics-server
maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
```

If we have only one non-master node, this results in one of the pods staying forever in the Pending state (because the only other node is the master, which is tainted and metrics-server doesn't have a respective toleration):

```
metrics-server-5c45c474f5-cb89k 1/1 Running 0 107m
metrics-server-5c45c474f5-t8ppf 0/1 Pending 0 107m
```

Since having a cluster with just one worker node and the metrics server addon enabled at the same time is a valid use case, such manifests that prevent the cluster from validating successfully (e.g., on kops rolling-update cluster) in this scenario, this should be considered a bug.

An ideal solution would be to make both the number of replicas and the maxSkew parameters configurable in the cluster spec. Less than ideal would allow to configure only one of them or hardcode relaxed topology spread constraints permanently and call it a day.

Another approach is to stop treating metrics-server pods as system-cluster-critical, because they aren't all that critical really.

Contributor guide

Open the contributing guide

Research direction

Start by examining the metrics-server addon deployment manifest and the cluster spec options controlling it. Reproduce validation with metrics-server enabled and fewer than two non-master nodes, then define and verify behavior for replica and topology-spread settings so the cluster can validate successfully in that configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.