self-hosted action runner with `kubernetes` mode on EKS failed at `Initialize containers` step from Action UI
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 1.4k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 24
Description
Describe the bug
ghcr.io/actions/actions-runner:2.317.0 self-hosted runner with kubernetes mode on EKS failed at Initialize containers step from Action UI.
To Reproduce
Steps to reproduce the behavior:
- Generate Runner deploy with the
values.yaml
runnerScaleSetName: "arc-runner"
githubConfigUrl: ####MASKED####
githubConfigSecret: ####MASKED####
containerMode:
type: "kubernetes"
kubernetesModeWorkVolumeClaim:
accessModes: ["ReadWriteOnce"]
storageClassName: "gp2"
resources:
requests:
storage: 2Gi
template:
spec:
initContainers:
- name: kube-init
image: ghcr.io/actions/actions-runner:2.317.0
command: ["sudo", "chown", "-R", "runner:runner", "/home/runner/_work"]
volumeMounts:
- name: work
mountPath: /home/runner/_work
containers:
- name: runner
image: ghcr.io/actions/actions-runner:2.317.0
command: ["/home/runner/run.sh"]
env:
- name: ACTIONS_RUNNER_CONTAINER_HOOKS
value: /home/runner/k8s/index.js
- name: ACTIONS_RUNNER_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
value: "false"
volumeMounts:
- name: work
mountPath: /home/runner/_work
nodeSelector:
karpenter.sh/capacity-type: spot
volumes:
- name: work
ephemeral:
volumeClaimTemplate:
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "gp2"
resources:
requests:
storage: 1Gi
- Set
.github/workflows/test.yaml
name: "[ARC] CI Test "
on:
pull_request:
jobs:
CI-Test:
permissions:
contents: read
issues: read
id-token: write
checks: write
pull-requests: write
runs-on: arc-runner
services:
mysql: .......
steps:
- uses: actions/checkout@v3
- name: Setup Python
run: |
sudo apt-get update
sudo apt-get install -y gcc python3-dev
- See error in
Initialize containersstep in Action UI. - Check
runnerpod log
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Starting process:
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] File name: '/home/runner/externals/node16/bin/node'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Arguments: '/home/runner/k8s/index.js'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Working directory: '/home/runner/_work/MASKED'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Require exit code zero: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Encoding web name: ; code page: ''
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Redirected STDIN: 'True'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Persist current code page: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Keep redirected STDIN open: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] High priority process: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Failed to update oom_score_adj for PID: 86.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] System.UnauthorizedAccessException: Access to the path '/proc/86/oom_score_adj' is denied.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] ---> System.IO.IOException: Permission denied
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] --- End of inner exception stack trace ---
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] at System.IO.Strategies.BufferedFileStreamStrategy.Dispose(Boolean disposing)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] at System.IO.StreamWriter.Dispose(Boolean disposing)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] at System.IO.File.WriteAllText(String path, String contents)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] at GitHub.Runner.Sdk.ProcessInvoker.WriteProcessOomScoreAdj(Int32 processId, Int32 oomScoreAdj)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Process started with process id 86, waiting for process exit.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Close STDIN after the first redirect finished.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] STDIN stream write finished.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Finished process 86 with exit code 1, and elapsed time 00:00:00.2572861.
[WORKER 2024-07-04 17:58:34Z ERR StepsRunner] Caught exception from step: System.Exception: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
[WORKER 2024-07-04 17:58:34Z ERR StepsRunner] ---> System.Exception: The hook script at '/home/runner/k8s/index.js' running command 'PrepareJob' did not execute successfully
[WORKER 2024-07-04 17:58:34Z ERR StepsRunner] at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
[WORKER 2024-07-04 17:58:34Z ERR StepsRunner] --- End of inner exception stack trace ---
[WORKER 2024-07-04 17:58:34Z ERR StepsRunner] at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
[WORKER 2024-07-04 17:58:34Z ERR StepsRunner] at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.PrepareJobAsync(IExecutionContext context, List`1 containers)
[WORKER 2024-07-04 17:58:34Z ERR StepsRunner] at GitHub.Runner.Worker.ContainerOperationProvider.StartContainersAsync(IExecutionContext executionContext, Object data)
[WORKER 2024-07-04 17:58:34Z ERR StepsRunner] at GitHub.Runner.Worker.JobExtensionRunner.RunAsync()
[WORKER 2024-07-04 17:58:34Z ERR StepsRunner] at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
The problem is "PrepareJob" step failed because of Access to the path '/proc/<PID>/oom_score_adj' is denied. even though I changed a permission on working directory. There were a similar issue already reported as a bug.
Expected behavior
kubernetes mode should work like dind mode.
Runner Version and Platform
Running on EKS
Runner: 2.317.0
What's not working?
Please include error messages and screenshots.
Job Log Output
Outputs from Action UI
##[debug]Evaluating condition for step: 'Initialize containers'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Initialize containers
##[debug]Register post job cleanup for stopping/deleting containers.
Run '/home/runner/k8s/index.js'
shell: /home/runner/externals/node16/bin/node {0}
##[debug]/home/runner/externals/node16/bin/node /home/runner/k8s/index.js
Error: Error: Job Container is required.
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug]System.Exception: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug] ---> System.Exception: The hook script at '/home/runner/k8s/index.js' running command 'PrepareJob' did not execute successfully
##[debug] at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug] --- End of inner exception stack trace ---
##[debug] at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug] at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.PrepareJobAsync(IExecutionContext context, List`1 containers)
##[debug] at GitHub.Runner.Worker.ContainerOperationProvider.StartContainersAsync(IExecutionContext executionContext, Object data)
##[debug] at GitHub.Runner.Worker.JobExtensionRunner.RunAsync()
##[debug] at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
##[debug]Finishing: Initialize containers
Runner and Worker's Diagnostic Logs
See above
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the /home/runner/k8s/index.js hook entry point and the logged GitHub.Runner.Sdk.ProcessInvoker.WriteProcessOomScoreAdj failure during PrepareJob. Reproduce the EKS Kubernetes-mode configuration and trace why the hook reports that a job container is required. Done means Kubernetes mode initializes containers successfully without the /proc//oom_score_adj permission failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, csharp, kubernetes
- Domain
- ci-cd, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100