aws / aws/eks-charts

aws-sigv4-proxy-admission-controller's webhook can result in entire cluster being inoperable

Open
#672 0 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Mustache
Stars
1.3k
Forks
1.1k
Avg merge
22m
Merged PRs (30d)
5

Description

**Describe the bug**
aws-sigv4-proxy-admission-controller's webhook doesn't handle outages well.

1. When webhook fails, it fails request (`failurePolicy` is not `Ignore`).
2. Webhook is triggered on controller itself and every other pod in cluster.

Both of these issues combined result in failure to start controller due to its own hook failing because controller is not ready. Circular dependency.

More than that, webhook is triggered when you add new nodes to the cluster. If controller is inoperable, this results in failed nodes (they are indefinitely 'not ready' and cannot join cluster). This in certain circumstances can result in entire cluster being inoperable.

**Steps to reproduce**
Install helm chart and depending on the order of manifests it will trigger.

It will also trigger if you delete all aws-sigv4-proxy-admission-controller instances from cluster or delete all cluster nodes and try creating them again.

**Expected outcome**
1. Controller deployment does not depend on its own hook.
2. Resources without sigv4 sidecar annotations do not require working aws-sigv4-proxy-admission-controller to deploy.

**Environment**

* Chart name: aws-sigv4-proxy-admission-controller
* Chart version: 0.1.2
* Kubernetes version: 1.20 and 1.21
* Using EKS (yes/no), if so version? 1.20 and 1.21

**Additional Context**:

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the aws-sigv4-proxy-admission-controller Helm chart, its webhook configuration, and the controller deployment. Reproduce installation-order and controller/node deletion scenarios on Kubernetes 1.20 or 1.21, then trace which resources match the webhook. Done means the controller does not trigger its own hook and unannotated resources can deploy when the controller is unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.