actions / actions/actions-runner-controller

Autoscaling Runner Sets not scaling down after long jobs

Open
#4,193 2 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug gha-runner-scale-set needs triage
Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Checks
Controller Version

0.12.1

Deployment Method

Kustomize

Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
The issue is happening intermittently, so I haven't been able to create a simple example to reliably reproduce it.
It seems to happen more often with long running jobs (>5 hours).
Describe the bug

When the job finishes successfully, the existing runner pod is removed. The EphemeralRunnerSet then immediately creates a new runner pod, as desired replicas is set to 1. This new runner pod remains idle until another workflow uses it. If no jobs request that runner set the idle runner remains indefinitely, despite minRunners being set to 0.

Describe the expected behavior

We set minRunners: 0 when deploying the gha-runner-scale-set chart, so we expect the EphemeralRunnerSet should scale down to 0 after the workflow finishes.

Additional Context
Listener logs
The listener never receives the message for the completed job.
https://gist.github.com/crystalthenR2R/d3c779ceb2bc14b9fec3a3e6f3fb5fa7

Running a new job on the runner immediately triggers a scale down:
https://gist.github.com/crystalthenR2R/870c6760f38bd2170e725681ba3c7288

Unfortunately this is happening with our larger sized runners with long/infrequent jobs. So they sometimes stick around for a very long time consuming a large amount of resources.
Controller Logs
Controller logs
https://gist.github.com/crystalthenR2R/a9e3010a78042178828146c81fa85454

The runner (nrxkx-runner-2cwzh) is marked as Finished at 2025-07-31 20:05:50, and a new runner is immediately created (nrxkx-runner-gfphq)
Runner Pod Logs
n/a

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the EphemeralRunnerSet lifecycle, the listener behavior, and the linked controller and listener logs; compare the long-job case with the case where a new job triggers scale-down. Verify the chart's minRunners: 0 configuration and determine why the completed job message is missed. Done means an idle runner set reliably scales from one runner to zero after a long workflow without requiring another job.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go, kubernetes
Domain
ci-cd, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.