aws / aws/amazon-eks-pod-identity-webhook

Injection sometimes fails on pods created from deployments

Open
#83 8 comments 6 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
691
Forks
202
Avg merge
4h 38m
Merged PRs (30d)
1

Description

**What happened**:

Volume and environment config was not injected into pods created from a deployment.

**What you expected to happen**:

The config to be injected, or some debug information about why the injection didn't happen.

**How to reproduce it (as minimally and precisely as possible)**:

Creating a couple of pods with a direct pod and a deployment:

```
(๑•ᴗ•)⊃━~/g/irsa-reproducer━☆゚ cat reproducer.yaml
apiVersion: v1
kind: Pod
metadata:
name: test-pod
namespace: test-app
spec:
serviceAccount: test-app
serviceAccountName: test-app
containers:
- name: hello
image: busybox
command: ['sh', '-c', 'sleep 3600']
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: test-app-token-njbrl
readOnly: true
volumes:
- name: test-app-token-njbrl
secret:
defaultMode: 420
secretName: test-app-token-njbrl
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-deploy
namespace: test-app
spec:
replicas: 1
selector:
matchLabels:
app: test-deploy
template:
metadata:
labels:
app: test-deploy
spec:
serviceAccount: test-app
serviceAccountName: test-app
containers:
- name: hello
image: busybox
command: ['sh', '-c', 'sleep 3600']
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: test-app-token-njbrl
readOnly: true
volumes:
- name: test-app-token-njbrl
secret:
defaultMode: 420
secretName: test-app-token-njbrl
```

And applying together:

```
(๑•ᴗ•)⊃━~/g/irsa-reproducer━☆゚ kubectl apply -f reproducer.yaml
```

Results in the directly created pod receiving the injected config, but sometimes not the pod created by the deployment:

```
(๑•ᴗ•)⊃━~/g/irsa-reproducer━☆゚ kubectl get pods -n test-app -o jsonpath='{range .items[*]}{@.metadata.name}{" (serviceAccount: "}{@.spec.serviceAccount}{") "}{@.spec.volumes[*].name}{"\n"}{end}'
test-deploy-6794999d9-cqkwm (serviceAccount: test-app) test-app-token-njbrl
test-pod (serviceAccount: test-app) aws-iam-token test-app-token-njbrl
```

**Anything else we need to know?**:

This happens much more regularity (almost guaranteed) on an EKS cluster in one of our AWS accounts, but not in other accounts.

Feels somewhat similar to https://github.com/godaddy/kubernetes-external-secrets/issues/419

**Environment**:
- AWS Region: eu-west-1
- EKS Platform version: `eks.2`
- Kubernetes version: `1.17`
- Webhook Version:

Contributor guide

Open the contributing guide

Research direction

Start with the reproducer.yaml manifest and run kubectl apply -f reproducer.yaml on the reported Kubernetes and EKS environments. Compare injection on the direct Pod with the Deployment-created Pod using the provided kubectl get pods command; done means the deployment pod receives the config consistently or provides useful debug information when injection is skipped.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.