aws / aws/amazon-eks-pod-identity-webhook
Cannot use wildcard (*) namespace in kops when using IRSA
- Dominant language
- Go
- Stars
- 691
- Forks
- 202
- Avg merge
- 4h 38m
- Merged PRs (30d)
- 1
Description
**What happened**:
We are trying to use wildcard namespace feature in kops that came up with this PR https://github.com/kubernetes/kops/pull/16113. Now using wildcard namespace in kops cluster manifest and then trying to create a pod that references the service account and IAM policy fails with this particular error in `pod-identity-webhook` logs:
```
I0821 08:42:55.226946 1 handler.go:395] Pod was not mutated. Reason: Service account did not have the right annotations or was not found in the cache. Pod=ssm-ec2-test, ServiceAccount=ssm-ec2, Namespace=default
```
**What you expected to happen**: Pod to be mutated and contain the required policy/role.
**How to reproduce it (as minimally and precisely as possible)**: in `kops` cluster manifest, we have this:
```
spec:
iam:
allowContainerRegistry: true
legacy: false
serviceAccountExternalPermissions:
- name: ssm-ec2
aws:
policyARNs:
- arn:aws:iam:::policy/access-ec2-with-ssm
namespace: "*"
```
Then we try to deploy an workload:
```
apiVersion: v1
kind: ServiceAccount
metadata:
name: ssm-ec2
namespace: default
---
apiVersion: v1
kind: Pod
metadata:
name: ssm-ec2-test
namespace: default
spec:
containers:
- name: aws-cli
image: amazon/aws-cli:latest
command:
- sleep
- "30000"
serviceAccountName: "ssm-ec2"
```
pod-identity-webhook complains with:
```
I0821 08:42:54.833148 1 cache.go:179] Adding SA default/ssm-ec2 to SA cache: &{RoleARN: Audience: UseRegionalSTS:false TokenExpiration:0}
I0821 08:42:54.833397 1 cache.go:179] Adding SA default/ssm-ec2 to SA cache: &{RoleARN: Audience: UseRegionalSTS:false TokenExpiration:0}
I0821 08:42:55.226659 1 cache.go:80] Fetching sa default/ssm-ec2 from cache
I0821 08:42:55.226847 1 cache.go:93] Service account default/ssm-ec2 not found in cache
I0821 08:42:55.226946 1 handler.go:395] Pod was not mutated. Reason: Service account did not have the right annotations or was not found in the cache. Pod=ssm-ec2-test, ServiceAccount=ssm-ec2, Namespace=default
```
**Anything else we need to know?**:
When we change the "*" to any namespace (default) everything works just fine as expected.
**Environment**:
- AWS Region: tested in eu-west-1 but should be valid in all.
- Kubernetes version (if using EKS, run `aws eks describe-cluster --name --query cluster.version`): 1.24.16 (not EKS)
- Webhook Version: `v0.4.0`
Contributor guide
Research direction
The failure is reported in cache.go and handler.go while processing a wildcard namespace from kops. Start by tracing how service-account entries are added to and fetched from the cache, comparing wildcard configuration with the working default-namespace case. Done means the reproduced pod is mutated with the expected IAM policy or role, with coverage for wildcard namespaces.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes
- Domain
- authentication, backend, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100