actions / actions/actions-runner-controller

Webhook ARC Scaling issue

Open
#2,306 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs triage
Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Checks
Controller Version

v0.27.0

Helm Chart Version

0.22.0

CertManager Version

1.5.3

Deployment Method

Helm

cert-manager installation

Yes

Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
  • I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
  • My actions-runner-controller version (v0.x.y) does support the feature
  • I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
  • I've migrated to the workflow job webhook event (if you using webhook driven scaling)
Resource Definitions
$ kubectl get runnerdeployment gha-staging-8c-32gb -n staging-8c-32gb -o yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  annotations:
    meta.helm.sh/release-name: staging-8c-32gb
    meta.helm.sh/release-namespace: staging-8c-32gb
  creationTimestamp: "2022-10-26T17:57:43Z"
  generation: 368232
  labels:
    app.kubernetes.io/managed-by: Helm
  name: gha-staging-8c-32gb
  namespace: staging-8c-32gb
  resourceVersion: "94653290"
  uid: 4c1cd628-8c0a-489a-bb6b-4475cae150db
spec:
  effectiveTime: "2023-02-21T23:07:20Z"
  replicas: 5
  selector: null
  template:
    metadata: {}
    spec:
      dockerVolumeMounts:
      - mountPath: /var/lib/docker
        name: docker-extra
      dockerdContainerResources: {}
      dockerdWithinRunnerContainer: true
      env:
      - name: RUNNER_FEATURE_FLAG_EPHEMERAL
        value: "true"
      - name: DISABLE_RUNNER_UPDATE
        value: "false"
      group: secondary-staging-8c-32gb
      image: prodweuadoacr.azurecr.io/github-runner-full:latest-stable
      imagePullSecrets:
      - name: artifactory-cred
      initContainers:
      - command:
        - sh
        - -c
        - chmod -R 606 /dev/kvm
        image: busybox
        name: kvm-permission
        resources: {}
        volumeMounts:
        - mountPath: /dev/kvm
          name: kvm-device
      labels:
      - staging-8c-32gb
      - weu
      - self-hosted
      - Linux
      - linux
      - X64
      - x64
      nodeSelector:
        agentpool: stgrunneri
      organization: tomtom-internal
      resources:
        limits:
          cpu: "8"
          ephemeral-storage: 250Gi
          memory: 32Gi
        requests:
          cpu: "8"
          ephemeral-storage: 250Gi
          memory: 32Gi
      sidecarContainers:
      - command:
        - bash
        - -c
        - --
        - bash -c 'while true; do tail -f /runner/_diag/*.log; [ $? -ne 0 ] && sleep
          3; done;'
        image: bash:latest
        name: logger
        resources: {}
        volumeMounts:
        - mountPath: /runner/_diag
          name: runnerlogs
      volumeMounts:
      - mountPath: /runner/_diag
        name: runnerlogs
      - mountPath: /dev/kvm
        name: kvm-device
      volumes:
      - hostPath:
          path: /mnt/docker-extra
          type: DirectoryOrCreate
        name: docker-extra
      - hostPath:
          path: /dev/kvm
        name: kvm-device
      - emptyDir: {}
        name: runnerlogs
status:
  availableReplicas: 6
  desiredReplicas: 5
  readyReplicas: 6
  replicas: 6
  updatedReplicas: 6

---
$ kubectl get hra gha-staging-8c-32gb-autoscaler -n staging-8c-32gb -o yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  annotations:
    meta.helm.sh/release-name: staging-8c-32gb
    meta.helm.sh/release-namespace: staging-8c-32gb
  creationTimestamp: "2023-02-02T15:06:09Z"
  generation: 41513
  labels:
    app.kubernetes.io/managed-by: Helm
  name: gha-staging-8c-32gb-autoscaler
  namespace: staging-8c-32gb
  resourceVersion: "94652004"
  uid: 9ac79c7a-135b-4802-a8ca-aa45e6f47f78
spec:
  capacityReservations:
  - effectiveTime: "2023-02-21T23:05:32Z"
    expirationTime: "2023-02-21T23:35:32Z"
    replicas: 1
  - effectiveTime: "2023-02-21T23:05:32Z"
    expirationTime: "2023-02-21T23:35:32Z"
    replicas: 1
  - effectiveTime: "2023-02-21T23:07:20Z"
    expirationTime: "2023-02-21T23:37:20Z"
    replicas: 1
  maxReplicas: 200
  minReplicas: 2
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    kind: RunnerDeployment
    name: gha-staging-8c-32gb
  scaleUpTriggers:
  - duration: 30m
    githubEvent:
      workflowJob: {}
status:
  desiredReplicas: 5
  lastSuccessfulScaleOutTime: "2023-02-21T23:07:20Z"

---
$ kubectl get storageclass
NAME                    PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
azurefile               file.csi.azure.com   Delete          Immediate              true                   118d
azurefile-csi           file.csi.azure.com   Delete          Immediate              true                   118d
azurefile-csi-premium   file.csi.azure.com   Delete          Immediate              true                   118d
azurefile-premium       file.csi.azure.com   Delete          Immediate              true                   118d
default (default)       disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   118d
managed                 disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   118d
managed-csi             disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   118d
managed-csi-premium     disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   118d
managed-premium         disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   118d
To Reproduce
1
Describe the bug

We are facing long queue times where workflow/job can not able to pick up a runner. I am not sure whether Failed to update runnerreplicaset resource and Runner pod is annotated to wait for completion, and the runner container is not restarting give some hints.

image

When we set manually or increase the minReplica, the workflow can able to pick up a runners as you can see from below picture, but I don't think we should manually manage this.

image

Here are some logs from controller.

2023-02-21T22:05:01Z	ERROR	runnerdeployment	Failed to update runnerreplicaset resource	{"runnerdeployment": "staging-8c-32gb/gha-staging-8c-32gb", "error": "Operation cannot be fulfilled on runnerreplicasets.actions.summerwind.dev \"gha-staging-8c-32gb-mplvl\": the object has been modified; please apply your changes to the latest version and try again"}
2023-02-21T22:05:01Z	ERROR	Reconciler error	{"controller": "runnerdeployment-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerDeployment", "RunnerDeployment": {"name":"gha-staging-8c-32gb","namespace":"staging-8c-32gb"}, "namespace": "staging-8c-32gb", "name": "gha-staging-8c-32gb", "reconcileID": "8c73a8ea-e026-427a-8d43-300cf18bd83e", "error": "Operation cannot be fulfilled on runnerreplicasets.actions.summerwind.dev \"gha-staging-8c-32gb-mplvl\": the object has been modified; please apply your changes to the latest version and try again"}
2023-02-21T22:18:25Z	ERROR	runnerreplicaset	Failed to patch owner to have actions-runner/unregistration-request-timestamp annotation	{"runnerreplicaset": "staging-05c-2gb/gha-staging-05c-2gb-zr4kl", "lastSyncTime": "2023-02-21T22:07:34Z", "effectiveTime": "2023-02-21 22:07:19 +0000 UTC", "templateHashDesired": "59bd66fdd9", "replicasDesired": 78, "replicasPending": 0, "replicasRunning": 82, "replicasMaybeRunning": 82, "templateHashObserved": ["59bd66fdd9"], "owner": "staging-05c-2gb/gha-staging-05c-2gb-zr4kl-g695w", "error": "Operation cannot be fulfilled on runners.actions.summerwind.dev \"gha-staging-05c-2gb-zr4kl-g695w\": the object has been modified; please apply your changes to the latest version and try again"}
2023-02-21T22:18:25Z	ERROR	Reconciler error	{"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "RunnerReplicaSet": {"name":"gha-staging-05c-2gb-zr4kl","namespace":"staging-05c-2gb"}, "namespace": "staging-05c-2gb", "name": "gha-staging-05c-2gb-zr4kl", "reconcileID": "e950f153-46ab-4ac8-aaff-b0ee11dd5f3f", "error": "Operation cannot be fulfilled on runners.actions.summerwind.dev \"gha-staging-05c-2gb-zr4kl-g695w\": the object has been modified; please apply your changes to the latest version and try again"}
2023-02-21T22:32:15Z	ERROR	runnerreplicaset	Failed to patch owner to have actions-runner/unregistration-request-timestamp annotation	{"runnerreplicaset": "staging-05c-2gb/gha-staging-05c-2gb-zr4kl", "lastSyncTime": "2023-02-21T22:29:28Z", "effectiveTime": "2023-02-21 22:07:19 +0000 UTC", "templateHashDesired": "59bd66fdd9", "replicasDesired": 75, "replicasPending": 0, "replicasRunning": 77, "replicasMaybeRunning": 77, "templateHashObserved": ["59bd66fdd9"], "owner": "staging-05c-2gb/gha-staging-05c-2gb-zr4kl-9jt2c", "error": "Operation cannot be fulfilled on runners.actions.summerwind.dev \"gha-staging-05c-2gb-zr4kl-9jt2c\": the object has been modified; please apply your changes to the latest version and try again"}
2023-02-21T22:32:15Z	ERROR	Reconciler error	{"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "RunnerReplicaSet": {"name":"gha-staging-05c-2gb-zr4kl","namespace":"staging-05c-2gb"}, "namespace": "staging-05c-2gb", "name": "gha-staging-05c-2gb-zr4kl", "reconcileID": "306f0438-ad62-42e0-9681-c979df6766d7", "error": "Operation cannot be fulfilled on runners.actions.summerwind.dev \"gha-staging-05c-2gb-zr4kl-9jt2c\": the object has been modified; please apply your changes to the latest version and try again"}
2023-02-21T22:46:05Z	ERROR	runnerreplicaset	Failed to patch owner to have actions-runner/unregistration-request-timestamp annotation	{"runnerreplicaset": "staging-05c-2gb/gha-staging-05c-2gb-zr4kl", "lastSyncTime": "2023-02-21T22:41:11Z", "effectiveTime": "2023-02-21 22:40:59 +0000 UTC", "templateHashDesired": "59bd66fdd9", "replicasDesired": 72, "replicasPending": 0, "replicasRunning": 74, "replicasMaybeRunning": 74, "templateHashObserved": ["59bd66fdd9"], "owner": "staging-05c-2gb/gha-staging-05c-2gb-zr4kl-7hp5c", "error": "Operation cannot be fulfilled on runners.actions.summerwind.dev \"gha-staging-05c-2gb-zr4kl-7hp5c\": the object has been modified; please apply your changes to the latest version and try again"}
2023-02-21T22:46:05Z	ERROR	Reconciler error	{"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "RunnerReplicaSet": {"name":"gha-staging-05c-2gb-zr4kl","namespace":"staging-05c-2gb"}, "namespace": "staging-05c-2gb", "name": "gha-staging-05c-2gb-zr4kl", "reconcileID": "4a7ff5f9-cb3b-4c1d-b304-162f0aca1623", "error": "Operation cannot be fulfilled on runners.actions.summerwind.dev \"gha-staging-05c-2gb-zr4kl-7hp5c\": the object has been modified; please apply your changes to the latest version and try again"}

2023-02-21T23:05:39Z	INFO	runnerpod	Runner pod is annotated to wait for completion, and the runner container is not restarting	{"runnerpod": "staging-8c-32gb/gha-staging-8c-32gb-mplvl-qsdn9"}
2023-02-21T23:05:39Z	INFO	runnerpod	Runner pod is annotated to wait for completion, and the runner container is not restarting	{"runnerpod": "staging-8c-32gb/gha-staging-8c-32gb-mplvl-qq4ft"}
2023-02-21T23:05:39Z	INFO	runnerpod	Runner pod is annotated to wait for completion, and the runner container is not restarting	{"runnerpod": "staging-8c-32gb/gha-staging-8c-32gb-mplvl-7v4sc"}
2023-02-21T23:05:40Z	INFO	runnerpod	Runner pod is annotated to wait for completion, and the runner container is not restarting	{"runnerpod": "staging-8c-32gb/gha-staging-8c-32gb-mplvl-zd2kh"}
...
Describe the expected behavior

Runner should be scaled up and scaled down properly.

Whole Controller Logs
https://gist.github.com/uralsemih/3eaf224add2fe6db5102aff738b78e2c
Whole Runner Pod Logs
https://gist.github.com/uralsemih/a237fd983236b8e724e8a5ca90bf3a0c
Additional Context

We have different RunnerDeployment and HRA per namespaces for different runner groups as below.

$ kubectl get ns
NAME                    STATUS   AGE
actions-runner-system   Active   118d
cert-manager            Active   118d
staging-05c-2gb         Active   118d
staging-1c-4gb          Active   118d
staging-2c-8gb          Active   118d
staging-4c-16gb         Active   118d
staging-8c-32gb         Active   118d
staging-8c-32gb-nav     Active   118d

$ kubectl get runnerdeployment --all-namespaces
NAMESPACE             NAME                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
staging-05c-2gb       gha-staging-05c-2gb       67        66        66           66          118d
staging-1c-4gb        gha-staging-1c-4gb        1         1         1            1           118d
staging-2c-8gb        gha-staging-2c-8gb        2         3         3            3           118d
staging-4c-16gb       gha-staging-4c-16gb       3         3         3            3           118d
staging-8c-32gb-nav   gha-staging-8c-32gb-nav   4         4         4            4           118d
staging-8c-32gb       gha-staging-8c-32gb       5         6         6            6           118d

$ kubectl get hra --all-namespaces
NAMESPACE             NAME                                 MIN   MAX   DESIRED   SCHEDULE
staging-05c-2gb       gha-staging-05c-2gb-autoscaler       1     200   67
staging-1c-4gb        gha-staging-1c-4gb-autoscaler        1     200   1
staging-2c-8gb        gha-staging-2c-8gb-autoscaler        1     200   2
staging-4c-16gb       gha-staging-4c-16gb-autoscaler       1     200   3
staging-8c-32gb-nav   gha-staging-8c-32gb-nav-autoscaler   1     30    4
staging-8c-32gb       gha-staging-8c-32gb-autoscaler       2     200   5

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the runnerdeployment-controller and runnerreplicaset-controller reconciliation paths referenced in the logs, using the supplied RunnerDeployment and HorizontalRunnerAutoscaler YAML to investigate the update conflicts. Done means queued workflow jobs consistently obtain runners without manually increasing minReplicas, and the reported resource-modification errors no longer block scaling.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go, helm, kubernetes
Domain
ci-cd, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.