Avoiding Deadlock in aws-sigv4-proxy-admission-controller Chart.
- Dominant language
- Mustache
- Stars
- 1.3k
- Forks
- 1.1k
- Avg merge
- 22m
- Merged PRs (30d)
- 5
Description
**Describe the bug**
A concise description of what the bug is.
I have a Cx deployed the chart and the node where aws-sigv4-proxy-admission-controller pod goes away, the hook locked it self out and any new ReplicaSet were not able to get provisioned since the proxy webhook it self was not able to get created with following error:
```
W0824 13:55:15.768558 11 dispatcher.go:182] Failed calling webhook, failing closed aws-sigv4-proxy-admission-controller.k8s.aws : failed calling webhook "aws-sigv4-proxy-admission-controller.k8s.aws ": Post "https://aws-sigv4-proxy-admission-controller-webhook-service.kube-system.svc:443/mutate?timeout=10s ": no endpoints available for service "aws-sigv4-proxy-admission-controller-webhook-service"
```
After mock up the environment I figured adding an objectSelector can effectively avoid this situation:
```
...
webhooks:
...
objectSelector:
matchExpressions:
- key: app
operator: NotIn
values:
- {{ template "aws-sigv4-proxy-admission-controller.fullname" . }}
...
...
```
As suggested by k8s official doc[2], please as well consider whether or not to avoid operating on the kube-system namespace in addition to adding the `objectSelector` as well, this can be achieved by adding `namespaceSelector` section under the webhook.
Reference:
[1] https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#avoiding-deadlocks-in-self-hosted-webhooks
[2] https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#avoiding-operating-on-the-kube-system-namespace
**Steps to reproduce**
1. Create a new EKS cluster
2. Install aws-sigv4-proxy-admission-controller chart
3. Delete the node where the webhook pod is running
4. Create any other deployment
You will find no new pods can be provisioned anymore. and the webhook's replica pod stucks in a deadlook since it requires itself to perform the mutating webhook validation.
**Expected outcome**
Inspect and merge #823
**Environment**
* Chart name: `aws-sigv4-proxy-admission-controller`
* Chart version: `0.1.2`
* Kubernetes version: `v1.22.11-eks-18ef993`
* Using EKS (yes/no), if so version? `yes, provided above`
Contributor guide
Research direction
Start with the aws-sigv4-proxy-admission-controller chart's MutatingWebhookConfiguration and compare the proposed objectSelector and namespaceSelector changes described in the issue and expected in #823. Reproduce the node-loss scenario, then verify that the webhook can recover without blocking its own pod or other workloads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100