aws / aws/amazon-eks-pod-identity-webhook

Pod Identity Association also subject to cache race condition?

Open
#264 5 comments 8 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
691
Forks
202
Avg merge
4h 38m
Merged PRs (30d)
1

Description

**What happened**:
Similar to #174 but specific to pod identity associations, we're observing the expected `AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE` env var is absent when a service account and pod are created within a short window. Typically we'll experience something like this:

- Programmatically create a pod identity association, service account annotated with IAM role, and pod in short succession
- The pod comes up, but AWS operations error with `An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: No OpenIDConnect provider found in your account for https://oidc.eks...`
- Examining the pod env, note that `AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE` is missing but `AWS_WEB_IDENTITY_TOKEN_FILE` is set
- Restart pod
- Note that `AWS_WEB_IDENTITY_TOKEN_FILE` is now replaced with `AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE` and pod operates as expected.

**What you expected to happen**:
If all the prerequisites are satisfied, pods should get the correct pod identity association mutation regardless of timing.

**How to reproduce it (as minimally and precisely as possible)**:
1. Create an IAM role with the correct pod identity association trust policy:
```yaml
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "pods.eks.amazonaws.com"
},
"Action": [
"sts:TagSession",
"sts:AssumeRole"
],
}
]
}
```
2. Create an EKS cluster, enabling the EKS Pod Identity Agent add-on.
3. Run `aws eks update-kubeconfig --name my-cluster`
4. Run:
```bash
$ aws eks create-pod-identity-association \
--cluster-name my-cluster \
--namespace default \
--service-account test-sa \
--role-arn arn:aws:iam::111111111111:role/test-role && \
sleep 0.75s && \
kubectl apply -f - <

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the timing-sensitive sequence with aws eks create-pod-identity-association followed by kubectl apply, then inspect pkg/containercredentials/config.go and its FileConfig entry point. Compare the behavior with issues #174, #236, and #252; done means newly created pods consistently receive AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE when the prerequisites are satisfied.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
authentication, cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.