Azure / Azure/azure-workload-identity
Cannot run proxy sidecar in an environment with hard ResourceQuotas enabled
- Dominant language
- Go
- Stars
- 338
- Forks
- 122
- Avg merge
- 1h 7m
- Merged PRs (30d)
- 3
Description
**Describe the bug**
The sidecar is entirely unusable with a namespace where a ResourceQuota with hard limits has been enabled. The webhook does not set or have any capabilities to set such limits as far as I can see from the code, making it impossible to start a process where quotas are enforced.
The code I'm referring to here:
- https://github.com/Azure/azure-workload-identity/blob/main/pkg/webhook/webhook.go#L190
- https://github.com/Azure/azure-workload-identity/blob/main/pkg/webhook/webhook.go#L220
**Steps To Reproduce**
Create a namespace and apply a ResourceQuota to it, for example:
```yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: resource-quota-name
namespace: resource-quota-namespace
spec:
hard:
count/configmaps: "100"
count/cronjobs.batch: "10"
count/deployments.apps: "100"
count/jobs.batch: "100"
count/persistentvolumeclaims: "10"
count/pods: "200"
count/services: "100"
limits.memory: 8Gi
requests.cpu: "6"
requests.memory: 8Gi
```
Finally, deploy a workload identity-enabled pod with resources set to this namespace, with the proxy enable using `azure.workload.identity/inject-proxy-sidecar: 'true'`. It should fail to start and complain about quota errors as mentioned below.
**Expected behavior**
The proxy and init container will start without issues.
**Logs**
```
message: 'pods "" is forbidden: failed quota:
: must specify limits.memory for: azwi-proxy,azwi-proxy-init; requests.cpu
for: azwi-proxy,azwi-proxy-init; requests.memory for: azwi-proxy,azwi-proxy-init'
```
**Environment**
- Kubernetes version (use `kubectl version`): v1.29.2
- Cloud provider or hardware configuration: AKS
- OS (e.g: `cat /etc/os-release`):
```
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
```
- Kernel (e.g. `uname -a`): `Linux 5.15.0-1068-azure #77-Ubuntu SMP Fri Jun 21 21:21:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux`
- Install tools: Terraform, ArgoCD, custom operators
- Network plugin and version (if this is a network-related bug): n/a
- Others: n/a
**Additional context**
Istio accomplishes a similar end using the `proxyCPU` and `proxyMemory` settings via annotations passed in on the pod - perhaps such an approach is applicable here?
- https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyCPU
- https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyCPULimit
- https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyMemory
- https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyMemoryLimit
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with pkg/webhook/webhook.go at the referenced lines and reproduce the failure using the ResourceQuota manifest and workload annotations in the issue. Trace how the proxy and init container are added, then determine how their resource settings should be supplied. Done means both injected containers start successfully in a namespace with hard quota requirements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100