actions / actions/actions-runner-controller

summerwind/actions-runner:latest not compatible with arm

Open
#2,047 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Checks
Controller Version

0.20.2

Helm Chart Version

No response

CertManager Version

No response

Deployment Method

ArgoCD

cert-manager installation

cert-manger installed with ArgoCD with helm chart.

  source:
    repoURL: https://charts.jetstack.io
    chart: cert-manager
    targetRevision: "v1.8.2"
Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
  • I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
  • My actions-runner-controller version (v0.x.y) does support the feature
  • I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
  • I've migrated to the workflow job webhook event (if you using webhook driven scaling)
Resource Definitions
Copied from Argo's "Live manifest".

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"actions.summerwind.dev/v1alpha1","kind":"RunnerDeployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"runner-vault-periodical-c6gn-4xlarge-dev","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"runner","app.kubernetes.io/version":"1.16.0","argocd.argoproj.io/instance":"runner-vault-periodical-c6gn-4xlarge-dev","helm.sh/chart":"runner-0.1.0"},"name":"runner-vault-periodical-c6gn-4xlarge-dev-deployment","namespace":"actions-runner-controller"},"spec":{"template":{"spec":{"labels":["vault-c6gn.4xlarge-dev"],"nodeSelector":{"node.kubernetes.io/instance-type":"c6gn.4xlarge"},"repository":"redact","serviceAccountName":"runner-vault"}}}}
  creationTimestamp: '2022-11-24T22:40:19Z'
  generation: 26
  labels:
    app.kubernetes.io/instance: runner-vault-periodical-c6gn-4xlarge-dev
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: runner
    app.kubernetes.io/version: 1.16.0
    argocd.argoproj.io/instance: runner-vault-periodical-c6gn-4xlarge-dev
    helm.sh/chart: runner-0.1.0
  name: runner-vault-periodical-c6gn-4xlarge-dev-deployment
  namespace: actions-runner-controller
  resourceVersion: '36436774'
  uid: f7387ae1-9171-4dd2-ab94-e95246698a95
spec:
  effectiveTime: null
  replicas: 1
  selector: null
  template:
    metadata: {}
    spec:
      dockerdContainerResources: {}
      image: ''
      labels:
        - vault-c6gn.4xlarge-dev
      nodeSelector:
        node.kubernetes.io/instance-type: c6gn.4xlarge
      repository: <redact>
      resources: {}
      serviceAccountName: runner-vault
status:
  availableReplicas: 1
  desiredReplicas: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
To Reproduce
1. Set up ARC RunnerDeployment on ARM based instance (such as `c6gn.4xlarge` on AWS)
2. It starts many pods that immediately terminate. See runner logs.

Anyway it looks like this:

~ » kubectl get po -n actions-runner-controller
NAME                                                              READY   STATUS              RESTARTS   AGE
actions-runner-controller-7fd69488dc-h7fcb                        2/2     Running             0          56s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-2cld7   1/2     Terminating         0          23s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-4bmd6   1/2     Terminating         0          21s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-4mvtv   1/2     Terminating         0          10s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-8bvs9   1/2     Terminating         0          14s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-98gst   1/2     Terminating         0          27s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-bbnfn   1/2     Terminating         0          6s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-gx4sx   1/2     Terminating         0          8s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-mm8g7   1/2     Terminating         0          25s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-mqx45   1/2     Terminating         0          19s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-qcc96   0/2     ContainerCreating   0          2s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-s9vcs   1/2     Terminating         0          33s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-srw58   1/2     Terminating         0          16s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-t7wdk   1/2     Terminating         0          4s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-tcpzv   1/2     Terminating         0          35s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-wprb7   1/2     Terminating         0          31s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-xplbb   1/2     Terminating         0          29s
runner-vault-periodical-c6gn-4xlarge-dev-deployment-ts4f2-zgbdb   1/2     Terminating         0          12s
Describe the bug

Seems like the :latest tag of summerwind/actions-runner have an x64 version of /usr/bin/dumb-init even for ARM images, which breaks it because of bad architecture.

This can be easily verified with:

~ » uname -m
arm64

~ » docker run -it --entrypoint=bash summerwind/actions-runner:latest                             
runner@147aad666fff:/$ file /usr/bin/dumb-init
/usr/bin/dumb-init: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped

In the meantime, as a workaround, I changed actionsRunnerRepositoryAndTag to summerwind/actions-runner:v2.298.2-ubuntu-20.04 (which is the older version).

Describe the expected behavior

It shouldn't break on ARM.

Whole Controller Logs
https://gist.github.com/nirhaas/ff856d4f7624476eac724db524e02b26
Whole Runner Pod Logs
Defaulted container "runner" out of: runner, docker
/usr/bin/entrypoint.sh: line 6: /usr/bin/dumb-init: cannot execute binary file: Exec format error
2022-11-25 02:11:46.999  NOTICE --- Runner init started with pid 7
2022-11-25 02:11:47.1  NOTICE --- Runner init exited. Exiting this process with code 0 so that the container and the pod is GC'ed Kubernetes soon.
Additional Context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the ARM failure with summerwind/actions-runner:latest and inspect /usr/bin/dumb-init and /usr/bin/entrypoint.sh. Compare the latest image with v2.298.2-ubuntu-20.04 and verify that the runner image's binaries use the ARM architecture; done means the pod starts successfully on an ARM node without an exec-format error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.