actions / actions/actions-runner-controller
0.14.2: AutoscalingListener can retain deleted EphemeralRunnerSet after runner template update
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 1.5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 27
Description
Checks
- I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
- I am using charts that are officially provided (rendered from the official 0.14.2 charts and applied through Argo CD).
Controller Version
0.14.2
Deployment Method
ArgoCD
Checks
- This isn't a question or user support case.
- I've read the changelog and this is not an expected backward-incompatible change.
To Reproduce
- Deploy
gha-runner-scale-set-controllerandgha-runner-scale-set0.14.2 with--update-strategy=eventual,minRunners: 1, andmaxRunners: 10. - Wait until the
AutoscalingRunnerSet,AutoscalingListener, andEphemeralRunnerSetare healthy. - Change
AutoscalingRunnerSet.spec.template.specso ARC rolls the EphemeralRunnerSet. In our case, a GitOps deployment removed an obsoleteruntimeClassNameand changed the runner container security context. - Observe that the old EphemeralRunnerSet is deleted and a different EphemeralRunnerSet survives, while
AutoscalingListener.spec.ephemeralRunnerSetNamestill names the deleted set. - The controller repeatedly recreates the listener pod from that unchanged
AutoscalingListener. Every pod attempts its initial patch against the deleted EphemeralRunnerSet, exits, and enters a permanent recreate/crash loop.
This happened during an update to an existing scale set, rather than a fresh install.
Describe the bug
On ARC 0.14.2, an AutoscalingListener remained bound to a deleted EphemeralRunnerSet after a runner-template update. The surviving EphemeralRunnerSet had a different generated name, but the listener CR, config, and RBAC still referenced the deleted name.
Every listener restart failed identically:
Application returned an error: handling initial message failed: could not patch ephemeral runner set,
patch JSON: {"spec":{"patchID":0,"replicas":1}},
error: ephemeralrunnersets.actions.github.com "<scale-set>-qb49n" not found
The controller then deleted and recreated the listener pod, but did not repair or replace the stale AutoscalingListener CR, so the loop continued indefinitely and jobs could not be assigned.
The symptoms and stale-reference mechanism match #4489 very closely. That issue covered 0.14.0/0.14.1 fresh installs and was closed by stale automation without a linked fix. This report confirms the same failure on 0.14.2 and during an update to an existing AutoscalingRunnerSet.
Recovery/workaround:
- Detect that
AutoscalingListener.spec.ephemeralRunnerSetNameis not the single active EphemeralRunnerSet. - Delete the stale AutoscalingListener CR.
- With
updateStrategy=eventual, ARC drains running jobs, creates the replacement EphemeralRunnerSet/listener, and recovers.
We currently run that check as a post-sync hook because the controller did not self-heal the stale reference.
Describe the expected behavior
After replacing an EphemeralRunnerSet, ARC should atomically create/update the listener against the surviving set, or reconcile and replace any listener whose spec.ephemeralRunnerSetName points to a missing/non-current set.
A listener pod restart should not repeatedly inherit a permanently stale generated resource name.
Additional Context
controllerVersion: 0.14.2
scaleSetChartVersion: 0.14.2
kubernetesVersion: v1.35.2
controllerArgs:
- --auto-scaling-runner-set-only
- --watch-single-namespace=<runner-namespace>
- --runner-max-concurrent-reconciles=2
- --update-strategy=eventual
scaleSet:
minRunners: 1
maxRunners: 10
observedState:
listenerTarget: <scale-set>-qb49n
listenerTargetExists: false
activeEphemeralRunnerSet: <different generated name>
relatedIssues:
- https://github.com/actions/actions-runner-controller/issues/4489
- https://github.com/actions/actions-runner-controller/issues/4432
Controller Logs
Complete captured controller log window (the controller includes the terminated listener's full output in message):
https://gist.github.com/DocX/80b533f37c0bd1a5253fdd8514df6609
The first entry already contains the decisive listener error; the recreate loop continues throughout the capture.
Runner Pod Logs
No runner pod failure was involved. The failing pod was the AutoscalingListener, and its complete output is embedded in the controller termination messages in the gist above. Runner creation/assignment was blocked because the listener could not patch its referenced EphemeralRunnerSet.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing reconciliation of AutoscalingRunnerSet, AutoscalingListener, and EphemeralRunnerSet during --update-strategy=eventual, using the reproduced runner-template update and the controller log gist. Done means a listener no longer retains a deleted EphemeralRunnerSet name and the listener pod recovers without manual CR deletion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100