Azure / Azure/azure-workload-identity

Cannot run proxy sidecar in an environment with hard ResourceQuotas enabled

Open
#1,441 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
338
Forks
122
Avg merge
1h 7m
Merged PRs (30d)
3

Description

**Describe the bug**
The sidecar is entirely unusable with a namespace where a ResourceQuota with hard limits has been enabled. The webhook does not set or have any capabilities to set such limits as far as I can see from the code, making it impossible to start a process where quotas are enforced.

The code I'm referring to here:
- https://github.com/Azure/azure-workload-identity/blob/main/pkg/webhook/webhook.go#L190
- https://github.com/Azure/azure-workload-identity/blob/main/pkg/webhook/webhook.go#L220

**Steps To Reproduce**
Create a namespace and apply a ResourceQuota to it, for example:
```yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: resource-quota-name
namespace: resource-quota-namespace
spec:
hard:
count/configmaps: "100"
count/cronjobs.batch: "10"
count/deployments.apps: "100"
count/jobs.batch: "100"
count/persistentvolumeclaims: "10"
count/pods: "200"
count/services: "100"
limits.memory: 8Gi
requests.cpu: "6"
requests.memory: 8Gi
```

Finally, deploy a workload identity-enabled pod with resources set to this namespace, with the proxy enable using `azure.workload.identity/inject-proxy-sidecar: 'true'`. It should fail to start and complain about quota errors as mentioned below.

**Expected behavior**
The proxy and init container will start without issues.

**Logs**
```
message: 'pods "" is forbidden: failed quota:
: must specify limits.memory for: azwi-proxy,azwi-proxy-init; requests.cpu
for: azwi-proxy,azwi-proxy-init; requests.memory for: azwi-proxy,azwi-proxy-init'
```

**Environment**

- Kubernetes version (use `kubectl version`): v1.29.2
- Cloud provider or hardware configuration: AKS
- OS (e.g: `cat /etc/os-release`):
```
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
```
- Kernel (e.g. `uname -a`): `Linux 5.15.0-1068-azure #77-Ubuntu SMP Fri Jun 21 21:21:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux`
- Install tools: Terraform, ArgoCD, custom operators
- Network plugin and version (if this is a network-related bug): n/a
- Others: n/a

**Additional context**
Istio accomplishes a similar end using the `proxyCPU` and `proxyMemory` settings via annotations passed in on the pod - perhaps such an approach is applicable here?
- https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyCPU
- https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyCPULimit
- https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyMemory
- https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyMemoryLimit

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with pkg/webhook/webhook.go at the referenced lines and reproduce the failure using the ResourceQuota manifest and workload annotations in the issue. Trace how the proxy and init container are added, then determine how their resource settings should be supplied. Done means both injected containers start successfully in a namespace with hard quota requirements.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go, kubernetes
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.