awslabs / awslabs/eks-perf-tests
[operator] Updating KIT Controlplane spec brings up all pods at once
- Dominant language
- Shell
- Stars
- 72
- Forks
- 50
- PR merge metrics
- No merged PRs in 30d
Description
When controlplane spec is updated with args, KIT operator recycles all of the System components like apiserver etc at once, this will lead to a outage during the update process.
See below all the apiserver pods for this controlplane are unavailable during update process:
```
dev-dsk-hakuna-2c-dfc2b72b % kubectl get pods | grep c0955120-57bc-420f-bec0-f79d28d86fbe
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b57t5d2 0/1 ContainerCreating 0 54s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b58v2wt 0/1 ContainerCreating 0 54s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b5zqqzg 0/1 ContainerCreating 0 54s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-bfnrm 1/1 Running 0 64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-gf4lp 0/1 Pending 0 64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-ms4vq 0/1 Pending 0 64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-0 1/1 Running 0 100m
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-1 0/1 Pending 0 5s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-2 0/1 Error 1 62s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787gwpl4 0/1 Pending 0 64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787m57zq 1/1 Running 0 64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787p82fz 0/1 Pending 0 64s
```
```
dev-dsk-hakuna-2c-dfc2b72b % kubectl get pods | grep c0955120-57bc-420f-bec0-f79d28d86fbe
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b57t5d2 1/1 Running 3 6m4s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b58v2wt 1/1 Running 3 6m4s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b5zqqzg 1/1 Running 3 6m4s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-bfnrm 1/1 Running 0 6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-gf4lp 1/1 Running 0 6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-ms4vq 1/1 Running 0 6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-0 1/1 Running 0 3m25s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-1 1/1 Running 2 5m15s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-2 1/1 Running 4 6m12s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787gwpl4 1/1 Running 0 6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787m57zq 1/1 Running 0 6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787p82fz 1/1 Running 0 6m14s
```
Expected: Pods should be updated based on rolling update strategy and should not disrupt current state of cluster.
Contributor guide
Research direction
Start by reproducing the controlplane spec update and inspect the affected pods with the provided kubectl get pods command. Trace the KIT operator's update behavior; done means apiserver and other System component pods use a rolling update and the cluster remains available throughout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, shell
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100