actions / actions/actions-runner-controller

HRA scale down ignores non-idling Runners

Open
#2,987 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug community needs triage
Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Checks
Controller Version

0.27.4

Helm Chart Version

0.23.3

CertManager Version

1.7.2

Deployment Method

Helm

cert-manager installation

yes

Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
  • I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
  • My actions-runner-controller version (v0.x.y) does support the feature
  • I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
  • I've migrated to the workflow job webhook event (if you using webhook driven scaling)
Resource Definitions
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: gh-xx
  namespace: ghrunner-xx
spec:
  template:
    spec:
      ephemeral: true
      labels:
      - Linux
      - X64
      - self-hosted
      - mylabel
      organization: myorg
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: gh-xx
  namespace: ghrunner-x
spec:
  maxReplicas: 10
  minReplicas: 0
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    kind: RunnerDeployment
    name: gh-xx
  scaleUpTriggers:
  - duration: 10m
    githubEvent:
      workflowJob: {}
To Reproduce
1. Start 1 Github Workflows which sleeps for 30 Minutes
2. Verify:
- Github UI: 1 running job
- HRA desired 1
- runnerdeployment desired 1, current 1, up-to-date 1, available 1
3. Wait 15 Minutes (time larger than hra.spec.scaleUpTriggers.duration):
- Github UI: 1 still running job
- HRA desired 0
- runnerdeployment desired 0, current 1, up-to-date 1, available 1
4. When you now trigger another workflow, then it will be in queuing state for 15 Minutes
- Github UI: 1 running and 1 queuing job
- HRA desired 1
- runnerdeployment desired 1, current 1, up-to-date 1, available 1
Describe the bug

I would expect that desired would include the currently non-idling runners.

Describe the expected behavior

No queuing time

Whole Controller Logs
na
Whole Runner Pod Logs
na
Additional Context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with the provided HorizontalRunnerAutoscaler and RunnerDeployment definitions on controller version 0.27.4, using a workflow that sleeps for 30 minutes. Trace the HRA scale-down decision when the running runner is non-idling and compare desired replicas with the active job count. Done means active runners remain covered by the desired count and a newly triggered workflow does not wait for scale-up.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go, kubernetes
Domain
ci-cd, devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.