actions / actions/actions-runner-controller

gha-runner-scale-set 0.14.2: deprecated-runner exit code 7 triggers scale-set delete/recreate loop; unbounded DeleteRunnerScaleSet call can wedge the controller permanently

Open
#4,600 3 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stale
Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Checks
  • Chart/controller version: gha-runner-scale-set-controller 0.14.2 (ghcr.io/actions/gha-runner-scale-set-controller:0.14.2), EKS 1.35, org-scoped scale set with runnerGroup, custom runner image based on ghcr.io/actions/actions-runner:2.334.0.
What happened (production incident, 2026-08-10)
  1. GitHub's rolling minimum runner version passed our pinned actions/runner 2.334.0. The service rejects the runner and the container exits with code 7, which ephemeralrunner_controller maps to phase Outdated ("Runner is deprecated").
  2. The Outdated phase escalates: EphemeralRunner → EphemeralRunnerSet (updateStatus: any outdated child ⇒ ERS Outdated) → AutoscalingRunnerSet (outdated at line ~231) → cleanUpResources tears down the listener, all runner sets, and deletes the GitHub-side runner scale set — then recreates everything, pulls the same stale image, and loops. A version-deadline event thus produces an infinite delete/recreate storm of the scale-set object itself.
  3. During that storm, one DeleteRunnerScaleSet call never returned. The Actions client sets no request deadline and the reconcile context is unbounded, so the single AutoscalingRunnerSet worker blocked for hours: pod Ready, zero log output, healthz green, no listener, 0 runners. Liveness cannot detect it. Restarting re-enters the same stuck delete (phase Outdated is persisted in status; its only exit is a successful cleanup).
  4. We additionally observed the stuck call did not honor HTTP(S)_PROXY env on the controller (other calls from the same manager, e.g. JIT-config, did traverse our egress proxy), so a proxy-enforced timeout could not bound it either.
Asks
  1. Deadlines on all Actions service calls (or reconcile-scoped timeouts) so a stalled call errors and requeues instead of starving the worker forever.
  2. Deprecated runner (exit 7) should not delete the GitHub-side scale set — tearing down and recreating the scale set cannot fix a stale image and amplifies API load; surfacing a terminal condition/event ("runner image below minimum version") would make the failure diagnosable in minutes instead of hours.
  3. A way out of persisted AutoscalingRunnerSetPhaseOutdated that does not require the GitHub-side delete to succeed.
  4. Reconcile-progress signal in /healthz (or a metric) so a wedged worker is restartable by liveness.
Reproduction sketch

Pin a runner image below the service minimum in a scale set with minRunners >= 1 and watch the Outdated teardown loop; any long-stalling DeleteRunnerScaleSet response then wedges the controller permanently.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the deprecated-runner path through ephemeralrunner_controller, EphemeralRunnerSet, AutoscalingRunnerSet, and cleanUpResources, including the AutoscalingRunnerSet logic around line 231 and the DeleteRunnerScaleSet call. Review how reconcile contexts, persisted Outdated status, and /healthz behave during a stalled request. Done means the listed failure modes are bounded, diagnosable, and covered by the project's relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go, kubernetes
Domain
ci-cd, devops, infrastructure, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.