actions / actions/actions-runner-controller
Pod startup failing due to new startup.sh
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 1.5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 27
Description
Controller Version
0.24.1
Helm Chart Version
0.19.1
CertManager Version
1.8.1
Deployment Method
Other
cert-manager installation
- Installation process with Helm Via flux. Tested and working correctly with custom domain.
Checks
- This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
- I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
- My actions-runner-controller version (v0.x.y) does support the feature
- I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
Resource Definitions
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
name: default-org-runners
namespace: actions-runner-system
spec:
replicas: 1
template:
spec:
image: summerwind/actions-runner-dind # <-- was unpinned
dockerdWithinRunnerContainer: true
organization: <redacted>
env: []
To Reproduce
Try to run on a GKE cluster with the latest image `latest@sha256:331d12b3c2bb35436f82dccfc661afd13bb093afd3461d19cc5fce74b64e896c`. Pod correctly schedules but `runner`
Describe the bug
See textPayload key below.
{
"textPayload": "tee: 'standard output': Bad file descriptor",
"insertId": "vrxebg7zrlowahpw",
"resource": {
"type": "k8s_container",
...
}
},
"timestamp": "2022-06-23T03:45:09.572895611Z",
"severity": "ERROR",
"labels": {
...
},
...
}
This is a k8s_container resource which means the pod was correctly scheduled but failed during startup.
Symptoms are a restart loop of pods every ~1 second.
Describe the expected behavior
Pod should start. Changing nothing but going back to summerwind/actions-runner-dind:v2.293.0-ubuntu-20.04-933b0c7@sha256:635aa33ed5fc83f5df7a27986f654500fc28eeb619498888f3442a133b54258b fixes the issue.
Controller Logs
https://gist.github.com/rich-bain/440988ac221992e196fba6aa4faeb711
Runner Pod Logs
Unavailable due to immediate deletion.
See stackdriver log (only a single item produced):
https://gist.github.com/rich-bain/440988ac221992e196fba6aa4faeb711
Additional Context
Worked fine yesterday. Spun up some new nodes which invalided my docker cache. Pinning back to the old version fixes the issue.
Issue either https://github.com/actions-runner-controller/actions-runner-controller/blob/master/runner/startup.sh#L30 or https://github.com/actions-runner-controller/actions-runner-controller/blob/master/runner/startup.sh#L45
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with runner/startup.sh, especially the suspected lines 30 and 45, and reproduce the failure using the latest image on a GKE cluster. Compare its startup behavior with the pinned v2.293.0-ubuntu-20.04 image and inspect the reported tee error. Done means the pod starts successfully without entering the one-second restart loop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, kubernetes, shell
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100