actions / actions/runner

self-hosted action runner with `kubernetes` mode on EKS failed at `Initialize containers` step from Action UI

Open
#3,372 6 comments 8 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Stale
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug
ghcr.io/actions/actions-runner:2.317.0 self-hosted runner with kubernetes mode on EKS failed at Initialize containers step from Action UI.

To Reproduce
Steps to reproduce the behavior:

  1. Generate Runner deploy with the values.yaml
runnerScaleSetName: "arc-runner"
githubConfigUrl: ####MASKED####
githubConfigSecret: ####MASKED####
  
containerMode:
  type: "kubernetes" 
  kubernetesModeWorkVolumeClaim:
    accessModes: ["ReadWriteOnce"]
    storageClassName: "gp2"
    resources:
      requests:
        storage: 2Gi

template:
  spec:
    initContainers: 
    - name: kube-init
      image: ghcr.io/actions/actions-runner:2.317.0
      command: ["sudo", "chown", "-R", "runner:runner", "/home/runner/_work"]
      volumeMounts:
      - name: work
        mountPath: /home/runner/_work
    containers:
    - name: runner
      image: ghcr.io/actions/actions-runner:2.317.0
      command: ["/home/runner/run.sh"]
      env:
        - name: ACTIONS_RUNNER_CONTAINER_HOOKS
          value: /home/runner/k8s/index.js
        - name: ACTIONS_RUNNER_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
          value: "false"
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
    nodeSelector:
      karpenter.sh/capacity-type: spot
    volumes:
      - name: work
        ephemeral:
          volumeClaimTemplate:
            spec:
              accessModes: [ "ReadWriteOnce" ]
              storageClassName: "gp2"
              resources:
                requests:
                  storage: 1Gi
  1. Set .github/workflows/test.yaml
name: "[ARC] CI Test "

on:
  pull_request:

jobs:
  CI-Test:
    permissions:
      contents: read
      issues: read
      id-token: write
      checks: write
      pull-requests: write
    runs-on: arc-runner
    services:
      mysql: .......
    steps:
      - uses: actions/checkout@v3
      - name: Setup Python
        run: |
          sudo apt-get update
          sudo apt-get install -y gcc python3-dev
  1. See error in Initialize containers step in Action UI.
  2. Check runner pod log
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Starting process:
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   File name: '/home/runner/externals/node16/bin/node'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Arguments: '/home/runner/k8s/index.js'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Working directory: '/home/runner/_work/MASKED'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Require exit code zero: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Encoding web name:  ; code page: ''
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Force kill process on cancellation: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Redirected STDIN: 'True'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Persist current code page: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Keep redirected STDIN open: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   High priority process: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Failed to update oom_score_adj for PID: 86.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] System.UnauthorizedAccessException: Access to the path '/proc/86/oom_score_adj' is denied.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]  ---> System.IO.IOException: Permission denied
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    --- End of inner exception stack trace ---
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    at System.IO.Strategies.BufferedFileStreamStrategy.Dispose(Boolean disposing)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    at System.IO.StreamWriter.Dispose(Boolean disposing)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    at System.IO.File.WriteAllText(String path, String contents)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    at GitHub.Runner.Sdk.ProcessInvoker.WriteProcessOomScoreAdj(Int32 processId, Int32 oomScoreAdj)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Process started with process id 86, waiting for process exit.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Close STDIN after the first redirect finished.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] STDIN stream write finished.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Finished process 86 with exit code 1, and elapsed time 00:00:00.2572861.
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner] Caught exception from step: System.Exception: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]  ---> System.Exception: The hook script at '/home/runner/k8s/index.js' running command 'PrepareJob' did not execute successfully
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]    at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]    --- End of inner exception stack trace ---
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]    at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]    at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.PrepareJobAsync(IExecutionContext context, List`1 containers)
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]    at GitHub.Runner.Worker.ContainerOperationProvider.StartContainersAsync(IExecutionContext executionContext, Object data)
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]    at GitHub.Runner.Worker.JobExtensionRunner.RunAsync()
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]    at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)

The problem is "PrepareJob" step failed because of Access to the path '/proc/<PID>/oom_score_adj' is denied. even though I changed a permission on working directory. There were a similar issue already reported as a bug.

Expected behavior
kubernetes mode should work like dind mode.

Runner Version and Platform

Running on EKS
Runner: 2.317.0

What's not working?

Please include error messages and screenshots.

Job Log Output

Outputs from Action UI

##[debug]Evaluating condition for step: 'Initialize containers'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Initialize containers
##[debug]Register post job cleanup for stopping/deleting containers.
Run '/home/runner/k8s/index.js'
  shell: /home/runner/externals/node16/bin/node {0}
##[debug]/home/runner/externals/node16/bin/node /home/runner/k8s/index.js
Error: Error: Job Container is required.
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug]System.Exception: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug] ---> System.Exception: The hook script at '/home/runner/k8s/index.js' running command 'PrepareJob' did not execute successfully
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug]   --- End of inner exception stack trace ---
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.PrepareJobAsync(IExecutionContext context, List`1 containers)
##[debug]   at GitHub.Runner.Worker.ContainerOperationProvider.StartContainersAsync(IExecutionContext executionContext, Object data)
##[debug]   at GitHub.Runner.Worker.JobExtensionRunner.RunAsync()
##[debug]   at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
##[debug]Finishing: Initialize containers

Runner and Worker's Diagnostic Logs

See above

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the /home/runner/k8s/index.js hook entry point and the logged GitHub.Runner.Sdk.ProcessInvoker.WriteProcessOomScoreAdj failure during PrepareJob. Reproduce the EKS Kubernetes-mode configuration and trace why the hook reports that a job container is required. Done means Kubernetes mode initializes containers successfully without the /proc//oom_score_adj permission failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, csharp, kubernetes
Domain
ci-cd, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.