actions / actions/actions-runner-controller

Pod startup failing due to new startup.sh

Open
#1,557 5 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Controller Version

0.24.1

Helm Chart Version

0.19.1

CertManager Version

1.8.1

Deployment Method

Other

cert-manager installation
  • Installation process with Helm Via flux. Tested and working correctly with custom domain.
Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
  • I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
  • My actions-runner-controller version (v0.x.y) does support the feature
  • I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
Resource Definitions
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: default-org-runners
  namespace: actions-runner-system
spec:
  replicas: 1
  template:
    spec:
      image: summerwind/actions-runner-dind # <-- was unpinned
      dockerdWithinRunnerContainer: true
      organization: <redacted>
      env: []
To Reproduce
Try to run on a GKE cluster with the latest image `latest@sha256:331d12b3c2bb35436f82dccfc661afd13bb093afd3461d19cc5fce74b64e896c`. Pod correctly schedules but `runner`
Describe the bug

See textPayload key below.

{
  "textPayload": "tee: 'standard output': Bad file descriptor",    
  "insertId": "vrxebg7zrlowahpw",
  "resource": {
    "type": "k8s_container",
    ...
    }
  },
  "timestamp": "2022-06-23T03:45:09.572895611Z",
  "severity": "ERROR",
  "labels": {
       ...
  },
   ...
}

This is a k8s_container resource which means the pod was correctly scheduled but failed during startup.

Symptoms are a restart loop of pods every ~1 second.

Describe the expected behavior

Pod should start. Changing nothing but going back to summerwind/actions-runner-dind:v2.293.0-ubuntu-20.04-933b0c7@sha256:635aa33ed5fc83f5df7a27986f654500fc28eeb619498888f3442a133b54258b fixes the issue.

Controller Logs
https://gist.github.com/rich-bain/440988ac221992e196fba6aa4faeb711
Runner Pod Logs
Unavailable due to immediate deletion.

See stackdriver log (only a single item produced):
https://gist.github.com/rich-bain/440988ac221992e196fba6aa4faeb711
Additional Context

Worked fine yesterday. Spun up some new nodes which invalided my docker cache. Pinning back to the old version fixes the issue.

Issue either https://github.com/actions-runner-controller/actions-runner-controller/blob/master/runner/startup.sh#L30 or https://github.com/actions-runner-controller/actions-runner-controller/blob/master/runner/startup.sh#L45

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with runner/startup.sh, especially the suspected lines 30 and 45, and reproduce the failure using the latest image on a GKE cluster. Compare its startup behavior with the pinned v2.293.0-ubuntu-20.04 image and inspect the reported tee error. Done means the pod starts successfully without entering the one-second restart loop.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes, shell
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.