aws / aws/amazon-eks-pod-identity-webhook
Injection sometimes fails on pods created from deployments
- Dominant language
- Go
- Stars
- 691
- Forks
- 202
- Avg merge
- 4h 38m
- Merged PRs (30d)
- 1
Description
**What happened**:
Volume and environment config was not injected into pods created from a deployment.
**What you expected to happen**:
The config to be injected, or some debug information about why the injection didn't happen.
**How to reproduce it (as minimally and precisely as possible)**:
Creating a couple of pods with a direct pod and a deployment:
```
(๑•ᴗ•)⊃━~/g/irsa-reproducer━☆゚ cat reproducer.yaml
apiVersion: v1
kind: Pod
metadata:
name: test-pod
namespace: test-app
spec:
serviceAccount: test-app
serviceAccountName: test-app
containers:
- name: hello
image: busybox
command: ['sh', '-c', 'sleep 3600']
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: test-app-token-njbrl
readOnly: true
volumes:
- name: test-app-token-njbrl
secret:
defaultMode: 420
secretName: test-app-token-njbrl
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-deploy
namespace: test-app
spec:
replicas: 1
selector:
matchLabels:
app: test-deploy
template:
metadata:
labels:
app: test-deploy
spec:
serviceAccount: test-app
serviceAccountName: test-app
containers:
- name: hello
image: busybox
command: ['sh', '-c', 'sleep 3600']
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: test-app-token-njbrl
readOnly: true
volumes:
- name: test-app-token-njbrl
secret:
defaultMode: 420
secretName: test-app-token-njbrl
```
And applying together:
```
(๑•ᴗ•)⊃━~/g/irsa-reproducer━☆゚ kubectl apply -f reproducer.yaml
```
Results in the directly created pod receiving the injected config, but sometimes not the pod created by the deployment:
```
(๑•ᴗ•)⊃━~/g/irsa-reproducer━☆゚ kubectl get pods -n test-app -o jsonpath='{range .items[*]}{@.metadata.name}{" (serviceAccount: "}{@.spec.serviceAccount}{") "}{@.spec.volumes[*].name}{"\n"}{end}'
test-deploy-6794999d9-cqkwm (serviceAccount: test-app) test-app-token-njbrl
test-pod (serviceAccount: test-app) aws-iam-token test-app-token-njbrl
```
**Anything else we need to know?**:
This happens much more regularity (almost guaranteed) on an EKS cluster in one of our AWS accounts, but not in other accounts.
Feels somewhat similar to https://github.com/godaddy/kubernetes-external-secrets/issues/419
**Environment**:
- AWS Region: eu-west-1
- EKS Platform version: `eks.2`
- Kubernetes version: `1.17`
- Webhook Version:
Contributor guide
Research direction
Start with the reproducer.yaml manifest and run kubectl apply -f reproducer.yaml on the reported Kubernetes and EKS environments. Compare injection on the direct Pod with the Deployment-created Pod using the provided kubectl get pods command; done means the deployment pod receives the config consistently or provides useful debug information when injection is skipped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100