actions / actions/runner

On self-hosted runners all workflow job failures report

Open
#3,178 3 comments 27 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug
When the CI job correctly returns a non-zero exit code (ie pylint find a problem) on a self-hosted runner the ContainerHookManager still requests users to contact their self-hosted runner administrator. I believe the error is coming from here https://github.com/actions/runner/blob/aa90563caecb8fcd2a1a84fc9c7833a56f6c40f4/src/Runner.Worker/Container/ContainerHooks/ContainerHookManager.cs#L177

To Reproduce
Steps to reproduce the behavior:

  1. We're using the GHA helm charts in kubernetes mode
  2. Run a CI job that exit 1 inside bash
  3. We get an error as in the attached screenshot (with debug logs) like
Error: Error: failed to run script step: command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/ccaaadc0-d72a-11ee-8141-5bd0b7d29442.sh], exit code 1
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.

Expected behavior
The ContainerHookManager should not consider a job returning non-zero exit codes an infrastructural failure. OR if there is a better way to for CI jobs to "fail" that should be better documented

Runner Version and Platform

Using the gha-scale-set-{,controller-}0.8.2 and ghcr.io/actions/actions-runner:2.312.0 all Linux

What's not working?

6Q3jxiaXZcXTFFH
Please include error messages and screenshots.

Job Log Output

Here are the debug logs of the exit 1 job

##[debug]Evaluating condition for step: 'Fail on purpose'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Fail on purpose
##[debug]Loading inputs
##[debug]Loading env
Run exit 1
##[debug]Overwrite 'working-directory' base on job defaults.
##[debug]Overwrite 'shell' base on job defaults.
##[debug]bash -e -c "source /root/.bashrc && bash -e /__w/_temp/2395bb1f-561b-438f-8af4-455aa2ba153e.sh"
Run '/home/runner/k8s/index.js'
##[debug]/home/runner/externals/node16/bin/node /home/runner/k8s/index.js
This bashrc in the pyle_env_dev container will set a venv and PYTHONPATH now
##[debug]{"message":"command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/6967fea0-d72c-11ee-982b-1ba0ed66fcbd.sh], exit code 1","details":{"causes":[{"reason":"ExitCode","message":"1"}]}}
##[debug]execPodStep failed: "command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/6967fea0-d72c-11ee-982b-1ba0ed66fcbd.sh], exit code 1"
Error: Error: failed to run script step: command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/6967fea0-d72c-11ee-982b-1ba0ed66fcbd.sh], exit code 1
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug]System.Exception: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug] ---> System.Exception: The hook script at '/home/runner/k8s/index.js' running command 'RunScriptStep' did not execute successfully
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug]   --- End of inner exception stack trace ---
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.RunScriptStepAsync(IExecutionContext context, ContainerInfo container, String workingDirectory, String entryPoint, String entryPointArgs, IDictionary`2 environmentVariables, String prependPath)
##[debug]   at GitHub.Runner.Worker.Handlers.ContainerStepHost.ExecuteAsync(IExecutionContext context, String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, String standardInInput, CancellationToken cancellationToken)
##[debug]   at GitHub.Runner.Worker.Handlers.ScriptHandler.RunAsync(ActionRunStage stage)
##[debug]   at GitHub.Runner.Worker.ActionRunner.RunAsync()
##[debug]   at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
##[debug]Finishing: Fail on purpose

Runner and Worker's Diagnostic Logs

I think the above covers all the involved logic.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at src/Runner.Worker/Container/ContainerHooks/ContainerHookManager.cs around line 177 and follow ExecuteHookScript and RunScriptStepAsync for the custom container hook path. Reproduce the issue with a self-hosted Kubernetes runner and a shell step using exit 1; done means a normal non-zero job exit is reported as a job failure without the infrastructure-failure administrator message.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.