actions / actions/actions-runner-controller
gha-runner-scale-set: no working way to protect in-flight jobs from node drain / cloud node auto-upgrade (PDBs non-functional, safe-to-evict doesn't apply to drains)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 1.5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 27
Description
What happened / the gap
A cloud node auto-upgrade (AKS) cordoned + drained the nodes running our gha-runner-scale-set runners mid-job. In-flight jobs died with ##[error]The runner has received a shutdown signal. A node drain is a voluntary eviction, so the expected protection is a PodDisruptionBudget — but there is currently no working pod-level mechanism to protect a busy ARC v2 runner from a drain:
- PDBs are non-functional on ARC v2. The runner pod's controller owner is the per-pod
EphemeralRunnerCR, which implements only astatussubresource, noscale. The K8s disruption controller resolves budgets via the pod's direct owner's scale subresource, can't find one, and marks the PDBSyncFailed(currentHealthy: 0, expectedPods: 0). Confirmed on controller0.14.2and still true onmaster(EphemeralRunnerCRDsubresources: {status: {}}). This was raised in #4493 and closed as architecturally won't-fix. cluster-autoscaler.kubernetes.io/safe-to-evictandkarpenter.sh/do-not-disruptdon't help — they're honored only by the Cluster Autoscaler / Karpenter when choosing to remove a node, not bykubectl drainor a cloud node auto-upgrade drain, which go through the eviction API and honor only PDBs. #2562 (busy-annotation request) is therefore CA-scoped and doesn't cover this.
Net: for the very common case of managed node auto-upgrade (AKS/EKS/GKE), there is no supported way to let a busy runner finish its job before its node is drained.
What I'd like
A supported mechanism to protect busy runners from voluntary node disruption. Options, roughly in order of preference:
- Make PDBs work — give
EphemeralRunnera scale subresource (reporting 1/1), or reparent runner pods to a set-level owner (EphemeralRunnerSet) that has one, so a standardPodDisruptionBudgetcan protect runner pods. This is the cleanest fit with how drains already work. (Reconsider #4493 with the node-drain use case, not just the API-compat framing.) - Native busy marker + graceful drain — controller sets a configurable label/annotation when a job lands and clears it on completion (#2562), and on receiving SIGTERM from a drain the runner finishes (or re-queues) the current job within
terminationGracePeriodSecondsinstead of dying immediately. - At minimum, document that ARC v2 runners cannot be protected from node drains and that the only mitigation is at the infrastructure layer (maintenance windows / controlled upgrade cadence).
Related: #4493 (PDB scale subresource, closed won't-fix), #2562 (busy annotations for CA), #4148 (EphemeralRunner stuck Running after node drain).
Environment
gha-runner-scale-set/gha-runner-scale-set-controller0.14.2- AKS, node auto-upgrade (drain via eviction API)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the EphemeralRunner CRD's status-only subresource and the controller behavior during node-drain termination; compare the findings with #4493, #2562, and #4148. Reproduce the AKS-style eviction scenario against controller 0.14.2 or master. Done requires an agreed, supported protection mechanism for busy runners, or explicit documentation of the infrastructure-only mitigation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, go, kubernetes
- Domain
- ci-cd, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100