actions / actions/actions-runner-controller

horizontalrunnerautoscaler Detected job with no labels, which is not supported by ARC. Skipping anyway

Open
#2,613 3 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs triage
Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Checks
Controller Version

v0.27.4

Helm Chart Version

0.23.3

CertManager Version

No response

Deployment Method

Helm

cert-manager installation

yes, this is the cert manager has been installed using

helm upgrade --install cert-manager jetstack/cert-manager \                                                                                                         
--namespace cert-manager \
--create-namespace \
--version v1.11.0 \
--set installCRDs=true --wait
Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
  • I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
  • My actions-runner-controller version (v0.x.y) does support the feature
  • I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
  • I've migrated to the workflow job webhook event (if you using webhook driven scaling)
Resource Definitions
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: self-hosted-large
  namespace: actions-runner-system
spec:
  template:
    metadata:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
    spec:
      serviceAccountName: github-actions-sa
      securityContext:
        # For Ubuntu 20.04 runner
        fsGroup: 1000
      organization: my-org
      image: summerwind/actions-runner-dind:latest
      imagePullPolicy: IfNotPresent
      ephemeral: true
      dockerEnabled: false
      dockerdWithinRunnerContainer: true
      containers:
      - name: runner
        resources:
          requests:
            memory: "10Gi"
            cpu: "3000m"
          limits:
            memory: "10Gi"
            cpu: "3000m"
      labels:
        - large
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  namespace: actions-runner-system
  name: self-hosted-large
spec:
  scaleDownDelaySecondsAfterScaleOut: 10
  scaleTargetRef:
    kind: RunnerDeployment
    name: self-hosted-large
  minReplicas: 0
  maxReplicas: 6
  metrics:
    - type: TotalNumberOfQueuedAndInProgressWorkflowRuns
      repositoryNames:
        - frontend
To Reproduce
this happens randomly, and the jobs have labels using this format:

runs-on: [self-hosted, large]

https://github.com/actions/actions-runner-controller/blob/032443fcfd4cf7b6e8bb09ed9dca639bcba9f8a4/controllers/actions.summerwind.net/autoscaling.go#L153



### Describe the bug

Randomly, the `horizontalrunnerautoscaler` doesn't update the desired replicas and the job waits indefinitely in github:

Requested labels: self-hosted, large
Job defined at: my-org/frontend/.github/workflows/zcommon_web_e2e_tests.yml@refs/heads/master
Reusable workflow chain:
my-org/frontend/.github/workflows/web_scheduled_e2e.yml@refs/heads/master (a9790cfa59ca77ead2f8ec4987a9cac8e98cfcce)
-> my-org/frontend/.github/workflows/zcommon_web_e2e_tests.yml@refs/heads/master (a9790cfa59ca77ead2f8ec4987a9cac8e98cfcce)
Waiting for a runner to pick up this job...

and the job uses the following label format

runs-on: [self-hosted, large]


should there be any dif between setting labels within quotes for the `horizontalrunnerautoscaler`?

runs-on: [self-hosted, large]


vs

runs-on: ["self-hosted", "large"]

?

any suggestion on how to further debug this?



### Describe the expected behavior

we shouldn't see this error in the ARC logs


### Whole Controller Logs

```shell
2023-05-22T10:02:22Z	INFO	horizontalrunnerautoscaler	Detected job with no labels, which is not supported by ARC. Skipping anyway.	{"labels": [], "run_id": 5044287443, "job_id": 13654547143}


### Whole Runner Pod Logs

```shell
there are no runner logs available
Additional Context

there is no runner in pending state, there are avialble resources on the node(s).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at controllers/actions.summerwind.net/autoscaling.go around line 153 and inspect how workflow job labels are read. Reproduce the issue with the supplied HorizontalRunnerAutoscaler, RunnerDeployment, and runs-on examples, then compare the queued job event with the logged empty labels. Done means supported labeled jobs no longer get skipped and the autoscaler updates desired replicas.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go, kubernetes
Domain
ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.