aws / aws/amazon-eks-pod-identity-webhook

ReInvocation not mutating for new containers in a pod

Open
#91 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
691
Forks
202
Avg merge
4h 38m
Merged PRs (30d)
1

Description

**What happened**:
We deployed a vault-agent-injector into our cluster which injects a vault-agent container into our pods via another mutating webhook.

When we deploy an application with the vault-agent annotations and a AWS annotation on the Service Account the main container has the AWS Environment vars injected as expected. Then the vault-agent-injector injects the vault-agent container into the pod as expected. However the AWS Environment vars aren't injected into this new container.

**What you expected to happen**:
We expect the AWS webhook to run again after vault-agent-injector webhook and inject the AWS Environment vars.

**How to reproduce it (as minimally and precisely as possible)**:
* Setup OIDC, SA with AWS role, etc (as normal)
* Setup vault-agent-injector (https://www.vaultproject.io/docs/platform/k8s/injector)
* Setup deployment with the following definition:
```
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: vault-aws-test
name: vault-aws-test
namespace: my-app
spec:
replicas: 1
selector:
matchLabels:
app: vault-aws-test
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: vault-aws-test
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-secret-foo-secret: secrets/my-app-secret
vault.hashicorp.com/agent-inject-status: update
vault.hashicorp.com/agent-pre-populate-only: "true"
spec:
serviceAccountName: my-app
containers:
- image: nginx
name: nginx
resources: {}
status: {}
```

**Anything else we need to know?**:
When looking at the events for mutations on the pod we see the following:
```
"mutation.webhook.admission.k8s.io/round_0_index_2": "{\"configuration\":\"pod-identity-webhook\",\"webhook\":\"iam-for-pods.amazonaws.com\",\"mutated\":true}",
"mutation.webhook.admission.k8s.io/round_0_index_4": "{\"configuration\":\"vault-sidecar-injector-agent-injector-cfg\",\"webhook\":\"vault.hashicorp.com\",\"mutated\":true}",
"mutation.webhook.admission.k8s.io/round_1_index_2": "{\"configuration\":\"pod-identity-webhook\",\"webhook\":\"iam-for-pods.amazonaws.com\",\"mutated\":false}",
```
However, we are unsure why the 2nd invocation of pod-identity-webhook didn't mutate the pod.

**Environment**:
- AWS Region: eu-west-2
- EKS Platform version (if using EKS, run `aws eks describe-cluster --name --query cluster.platformVersion`):
eks.2
- Kubernetes version (if using EKS, run `aws eks describe-cluster --name --query cluster.version`):
1.18
- Webhook Version:

Contributor guide

Open the contributing guide

Research direction

Start with the pod-identity-webhook entry point and reproduce the supplied Deployment alongside the vault-agent-injector. Inspect the Kubernetes mutation events, especially the round_1 invocation marked mutated:false, and determine why the injected container lacks AWS environment variables. Done means the reproduction is explained and the new container receives the expected variables after reinvocation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.