actions / actions/runner-container-hooks
kubernetes-novolume mode: IRSA projected volume conflicts with container hooks mkdir
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 153
- Forks
- 112
- Avg merge
- 6m
- Merged PRs (30d)
- 1
Description
Description
When using containerMode.type: kubernetes-novolume with an IRSA annotation service account on worker pods, the container hook fails with:
mkdir: cannot create directory '/var/run/secrets/eks.amazonaws.com/serviceaccount/token': File exists
Root Cause
The container hooks detect the runner pod's AWS_WEB_IDENTITY_TOKEN_FILE env var which is set by the EKS IRSA webhook, and attempts to replicate the path in the worker pod. The hooks set AWS_SERVICE_ACCOUNT_VOLUME to the full token file path (/var/run/secrets/eks.amazonaws.com/serviceaccount/token) and runs mkdir -p on it via kubectl exec.
However, when the worker pod also uses an IRSA annotated service account, the EKS IRSA webhook has already injected a volume that mounts token as a file at that path. This leads mkdir to fail because a file already exists where the hooks expect to create a directory.
When describing one of these worker pods you can see the following:
Init Containers:
fs-init:
State: Terminated
Reason: Completed # <-- init container succeeds
Exit Code: 0
Environment:
AWS_SERVICE_ACCOUNT_VOLUME: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
...
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
Volumes:
aws-iam-token:
Type: Projected # <-- IRSA webhook injected this
The fs-init container completes successfully, the mkdir error occurs after the pod is running during the exec file operations the hooks perform to set up the worker pod's environment.
Steps to reproduce
- Deploy a runner on EKS with
containerMode.type: kubernetes-novolume - Create a ServiceAccount with the IRSA annotation:
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-runner-sa
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-role
- Configure the runner pod template to use this ServiceAccount
- Configure a hook template that sets the same ServiceAccount on worker pods:
spec: serviceAccountName: my-runner-sa - Run a workflow — the worker pod creates and starts successfully, but the hooks fail during exec-based file setup:
(node:76) [DEP0005] DeprecationWarning: Buffer() is deprecated... mkdir: cannot create directory '/var/run/secrets/eks.amazonaws.com/serviceaccount/token': File exists Error: Error: command terminated with non-zero exit code: command terminated with exit code 1 Error: Executing the custom container implementation failed.
This only occurns in kubernetes-novolume, not in kubernetes.
Environment
- Container hooks version: v0.8.1
- Actions/Runner container version: v2.334.0
- ARC chart version: v0.14.1
- Container mode:
kubernetes-novolume - Kubernetes: Amazon EKS with the EKS Pod Identity Webhook
Expected Behavior
Worker pods should init successfully when their service account has an IRSA annotation. The container hooks should handle the case where the IRSA projected volume has already mounted token as a file at the path the hooks attempt to mkdir.
Actual Behavior
The hooks treat the full AWS_WEB_IDENTITY_TOKEN_FILE path (ending in the token file) as a directory and run mkdir -p on it. Since the IRSA webhook already mounted token as a file via a projected volume, mkdir fails.
Potential Temporary Workaround
You can create a separate service account (ex: worker-sa) for the worker pod that doesn't have an IRSA annotation. This will allow the worker container to have access to whatever RBAC permissions are in worker-sa, but it won't have any AWS permissions. Workflows that need AWS access would need an alternate mechanism, such as aws-actions/configure-aws-credentials with GitHub OIDC.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how the kubernetes-novolume container hooks derive and use AWS_SERVICE_ACCOUNT_VOLUME during their exec-based file setup. Reproduce the issue with an IRSA-annotated ServiceAccount and inspect the worker pod's projected aws-iam-token volume. Done means the hooks handle a token file already mounted at the target path and the workflow completes without the mkdir failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, typescript
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100