awslabs / awslabs/eks-perf-tests

[operator] Updating KIT Controlplane spec brings up all pods at once

Open
#79 0 comments 0 reactions 0 assignees View on GitHub
0.2 operator
Dominant language
Shell
Stars
72
Forks
50
PR merge metrics
No merged PRs in 30d

Description

When controlplane spec is updated with args, KIT operator recycles all of the System components like apiserver etc at once, this will lead to a outage during the update process.

See below all the apiserver pods for this controlplane are unavailable during update process:

```
dev-dsk-hakuna-2c-dfc2b72b % kubectl get pods | grep c0955120-57bc-420f-bec0-f79d28d86fbe
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b57t5d2 0/1 ContainerCreating 0 54s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b58v2wt 0/1 ContainerCreating 0 54s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b5zqqzg 0/1 ContainerCreating 0 54s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-bfnrm 1/1 Running 0 64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-gf4lp 0/1 Pending 0 64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-ms4vq 0/1 Pending 0 64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-0 1/1 Running 0 100m
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-1 0/1 Pending 0 5s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-2 0/1 Error 1 62s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787gwpl4 0/1 Pending 0 64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787m57zq 1/1 Running 0 64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787p82fz 0/1 Pending 0 64s

```

```
dev-dsk-hakuna-2c-dfc2b72b % kubectl get pods | grep c0955120-57bc-420f-bec0-f79d28d86fbe
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b57t5d2 1/1 Running 3 6m4s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b58v2wt 1/1 Running 3 6m4s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b5zqqzg 1/1 Running 3 6m4s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-bfnrm 1/1 Running 0 6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-gf4lp 1/1 Running 0 6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-ms4vq 1/1 Running 0 6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-0 1/1 Running 0 3m25s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-1 1/1 Running 2 5m15s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-2 1/1 Running 4 6m12s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787gwpl4 1/1 Running 0 6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787m57zq 1/1 Running 0 6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787p82fz 1/1 Running 0 6m14s
```

Expected: Pods should be updated based on rolling update strategy and should not disrupt current state of cluster.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the controlplane spec update and inspect the affected pods with the provided kubectl get pods command. Trace the KIT operator's update behavior; done means apiserver and other System component pods use a rolling update and the cluster remains available throughout.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, shell
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.