aws / aws/amazon-eks-pod-identity-webhook
Docker images with non-root account fails to read token file
- Dominant language
- Go
- Stars
- 691
- Forks
- 202
- Avg merge
- 4h 38m
- Merged PRs (30d)
- 1
Description
**What happened**:
Tried to install External DNS to my EKS cluster with amazon-eks-pod-identity-webhook
**What you expected to happen**:
External DNS working with IAM credentials provided by amazon-eks-pod-identity-webhook
**How to reproduce it (as minimally and precisely as possible)**:
1. [Enable IAM Roles for Service Accounts on your Cluster](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html)
2. Setup amazon-eks-pod-identity-webhook to the cluster.
3. Use the [latest master branch of External DNS](https://github.com/kubernetes-incubator/external-dns/commit/8da3b34f6d7cdd290a42311b42fbd01cf3b98e3e)
4. Annotate the external-dns service account:
```
apiVersion: v1
kind: ServiceAccount
metadata:
name: external-dns
namespace: external-dns
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam:::role/K8sExternalDNSRole
```
5. External DNS pod logs show:
```
time="2019-09-11T07:31:53Z" level=error msg="WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token\ncaused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied"
```
Note: I confirmed amazon-eks-pod-identity-webhook is working properly when using root accounts in the pods.
**Anything else we need to know?**:
I've reported this issue to external dns project: kubernetes-incubator/external-dns#1185
And it seems odd that kubernetes service account volume mounted token files are with permission 0644 and the EKS tokens are with permission 0600.
EKS tokens seems to be 0600:
```
~ $ ls -al /var/run/secrets/eks.amazonaws.com/serviceaccount/
total 0
drwxrwxrwt 3 root root 100 Sep 11 06:40 .
drwxr-xr-x 3 root root 28 Sep 11 06:40 ..
drwxr-xr-x 2 root root 60 Sep 11 06:40 ..2019_09_11_06_40_49.865776187
lrwxrwxrwx 1 root root 31 Sep 11 06:40 ..data -> ..2019_09_11_06_40_49.865776187
lrwxrwxrwx 1 root root 12 Sep 11 06:40 token -> ..data/token
~ $ ls -al /var/run/secrets/eks.amazonaws.com/serviceaccount/..data/token
-rw------- 1 root root 1028 Sep 11 06:40 /var/run/secrets/eks.amazonaws.com/serviceaccount/..data/token
```
Kubernetes tokens seems to be 0644:
```
/run/secrets/kubernetes.io/serviceaccount $ ls -al
total 0
drwxrwxrwt 3 root root 140 Sep 11 07:31 .
drwxr-xr-x 3 root root 28 Sep 11 07:31 ..
drwxr-xr-x 2 root root 100 Sep 11 07:31 ..2019_09_11_07_31_49.116680770
lrwxrwxrwx 1 root root 31 Sep 11 07:31 ..data -> ..2019_09_11_07_31_49.116680770
lrwxrwxrwx 1 root root 13 Sep 11 07:31 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root 16 Sep 11 07:31 namespace -> ..data/namespace
lrwxrwxrwx 1 root root 12 Sep 11 07:31 token -> ..data/token
/run/secrets/kubernetes.io/serviceaccount $ ls -al ..data/
total 12
drwxr-xr-x 2 root root 100 Sep 11 07:31 .
drwxrwxrwt 3 root root 140 Sep 11 07:31 ..
-rw-r--r-- 1 root root 1025 Sep 11 07:31 ca.crt
-rw-r--r-- 1 root root 12 Sep 11 07:31 namespace
-rw-r--r-- 1 root root 875 Sep 11 07:31 token
```
I've seen the code, and the code doesn't specify the token file permission. So the default 0644 should be used. Maybe this is a k8s upstream bug?
**Environment**:
- AWS Region: ap-northeast-1
- EKS Platform version (if using EKS, run `aws eks describe-cluster --name --query cluster.platformVersion`): "eks.1"
- Kubernetes version (if using EKS, run `aws eks describe-cluster --name --query cluster.version`): "1.14"
- Webhook Version: master (Commit ID: d2d603977e19b47388d913d640828418e173d3e0)
Contributor guide
Research direction
Start by examining the amazon-eks-pod-identity-webhook behavior that creates the EKS service-account token and reproduce the permission-denied error in a non-root container. Compare the EKS token permissions with the Kubernetes service-account volume behavior described in the issue and review the linked external-dns report. Done means the documented non-root External DNS scenario can read its token without regressing credential security.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, go
- Domain
- authentication, cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100