On self-hosted runners all workflow job failures report
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 1.4k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 24
Description
Describe the bug
When the CI job correctly returns a non-zero exit code (ie pylint find a problem) on a self-hosted runner the ContainerHookManager still requests users to contact their self-hosted runner administrator. I believe the error is coming from here https://github.com/actions/runner/blob/aa90563caecb8fcd2a1a84fc9c7833a56f6c40f4/src/Runner.Worker/Container/ContainerHooks/ContainerHookManager.cs#L177
To Reproduce
Steps to reproduce the behavior:
- We're using the GHA helm charts in kubernetes mode
- Run a CI job that
exit 1inside bash - We get an error as in the attached screenshot (with debug logs) like
Error: Error: failed to run script step: command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/ccaaadc0-d72a-11ee-8141-5bd0b7d29442.sh], exit code 1
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
Expected behavior
The ContainerHookManager should not consider a job returning non-zero exit codes an infrastructural failure. OR if there is a better way to for CI jobs to "fail" that should be better documented
Runner Version and Platform
Using the gha-scale-set-{,controller-}0.8.2 and ghcr.io/actions/actions-runner:2.312.0 all Linux
What's not working?
Please include error messages and screenshots.
Job Log Output
Here are the debug logs of the exit 1 job
##[debug]Evaluating condition for step: 'Fail on purpose'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Fail on purpose
##[debug]Loading inputs
##[debug]Loading env
Run exit 1
##[debug]Overwrite 'working-directory' base on job defaults.
##[debug]Overwrite 'shell' base on job defaults.
##[debug]bash -e -c "source /root/.bashrc && bash -e /__w/_temp/2395bb1f-561b-438f-8af4-455aa2ba153e.sh"
Run '/home/runner/k8s/index.js'
##[debug]/home/runner/externals/node16/bin/node /home/runner/k8s/index.js
This bashrc in the pyle_env_dev container will set a venv and PYTHONPATH now
##[debug]{"message":"command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/6967fea0-d72c-11ee-982b-1ba0ed66fcbd.sh], exit code 1","details":{"causes":[{"reason":"ExitCode","message":"1"}]}}
##[debug]execPodStep failed: "command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/6967fea0-d72c-11ee-982b-1ba0ed66fcbd.sh], exit code 1"
Error: Error: failed to run script step: command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/6967fea0-d72c-11ee-982b-1ba0ed66fcbd.sh], exit code 1
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug]System.Exception: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug] ---> System.Exception: The hook script at '/home/runner/k8s/index.js' running command 'RunScriptStep' did not execute successfully
##[debug] at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug] --- End of inner exception stack trace ---
##[debug] at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug] at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.RunScriptStepAsync(IExecutionContext context, ContainerInfo container, String workingDirectory, String entryPoint, String entryPointArgs, IDictionary`2 environmentVariables, String prependPath)
##[debug] at GitHub.Runner.Worker.Handlers.ContainerStepHost.ExecuteAsync(IExecutionContext context, String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, String standardInInput, CancellationToken cancellationToken)
##[debug] at GitHub.Runner.Worker.Handlers.ScriptHandler.RunAsync(ActionRunStage stage)
##[debug] at GitHub.Runner.Worker.ActionRunner.RunAsync()
##[debug] at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
##[debug]Finishing: Fail on purpose
Runner and Worker's Diagnostic Logs
I think the above covers all the involved logic.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at src/Runner.Worker/Container/ContainerHooks/ContainerHookManager.cs around line 177 and follow ExecuteHookScript and RunScriptStepAsync for the custom container hook path. Reproduce the issue with a self-hosted Kubernetes runner and a shell step using exit 1; done means a normal non-zero job exit is reported as a job failure without the infrastructure-failure administrator message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100