aws / aws/amazon-eks-pod-identity-webhook
ReInvocation not mutating for new containers in a pod
- Dominant language
- Go
- Stars
- 691
- Forks
- 202
- Avg merge
- 4h 38m
- Merged PRs (30d)
- 1
Description
**What happened**:
We deployed a vault-agent-injector into our cluster which injects a vault-agent container into our pods via another mutating webhook.
When we deploy an application with the vault-agent annotations and a AWS annotation on the Service Account the main container has the AWS Environment vars injected as expected. Then the vault-agent-injector injects the vault-agent container into the pod as expected. However the AWS Environment vars aren't injected into this new container.
**What you expected to happen**:
We expect the AWS webhook to run again after vault-agent-injector webhook and inject the AWS Environment vars.
**How to reproduce it (as minimally and precisely as possible)**:
* Setup OIDC, SA with AWS role, etc (as normal)
* Setup vault-agent-injector (https://www.vaultproject.io/docs/platform/k8s/injector)
* Setup deployment with the following definition:
```
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: vault-aws-test
name: vault-aws-test
namespace: my-app
spec:
replicas: 1
selector:
matchLabels:
app: vault-aws-test
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: vault-aws-test
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-secret-foo-secret: secrets/my-app-secret
vault.hashicorp.com/agent-inject-status: update
vault.hashicorp.com/agent-pre-populate-only: "true"
spec:
serviceAccountName: my-app
containers:
- image: nginx
name: nginx
resources: {}
status: {}
```
**Anything else we need to know?**:
When looking at the events for mutations on the pod we see the following:
```
"mutation.webhook.admission.k8s.io/round_0_index_2": "{\"configuration\":\"pod-identity-webhook\",\"webhook\":\"iam-for-pods.amazonaws.com\",\"mutated\":true}",
"mutation.webhook.admission.k8s.io/round_0_index_4": "{\"configuration\":\"vault-sidecar-injector-agent-injector-cfg\",\"webhook\":\"vault.hashicorp.com\",\"mutated\":true}",
"mutation.webhook.admission.k8s.io/round_1_index_2": "{\"configuration\":\"pod-identity-webhook\",\"webhook\":\"iam-for-pods.amazonaws.com\",\"mutated\":false}",
```
However, we are unsure why the 2nd invocation of pod-identity-webhook didn't mutate the pod.
**Environment**:
- AWS Region: eu-west-2
- EKS Platform version (if using EKS, run `aws eks describe-cluster --name --query cluster.platformVersion`):
eks.2
- Kubernetes version (if using EKS, run `aws eks describe-cluster --name --query cluster.version`):
1.18
- Webhook Version:
Contributor guide
Research direction
Start with the pod-identity-webhook entry point and reproduce the supplied Deployment alongside the vault-agent-injector. Inspect the Kubernetes mutation events, especially the round_1 invocation marked mutated:false, and determine why the injected container lacks AWS environment variables. Done means the reproduction is explained and the new container receives the expected variables after reinvocation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes
- Domain
- cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100