actions / actions/actions-runner-controller

gha-runner-scale-set: no working way to protect in-flight jobs from node drain / cloud node auto-upgrade (PDBs non-functional, safe-to-evict doesn't apply to drains)

Open
#4,618 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

What happened / the gap

A cloud node auto-upgrade (AKS) cordoned + drained the nodes running our gha-runner-scale-set runners mid-job. In-flight jobs died with ##[error]The runner has received a shutdown signal. A node drain is a voluntary eviction, so the expected protection is a PodDisruptionBudget — but there is currently no working pod-level mechanism to protect a busy ARC v2 runner from a drain:

  1. PDBs are non-functional on ARC v2. The runner pod's controller owner is the per-pod EphemeralRunner CR, which implements only a status subresource, no scale. The K8s disruption controller resolves budgets via the pod's direct owner's scale subresource, can't find one, and marks the PDB SyncFailed (currentHealthy: 0, expectedPods: 0). Confirmed on controller 0.14.2 and still true on master (EphemeralRunner CRD subresources: {status: {}}). This was raised in #4493 and closed as architecturally won't-fix.
  2. cluster-autoscaler.kubernetes.io/safe-to-evict and karpenter.sh/do-not-disrupt don't help — they're honored only by the Cluster Autoscaler / Karpenter when choosing to remove a node, not by kubectl drain or a cloud node auto-upgrade drain, which go through the eviction API and honor only PDBs. #2562 (busy-annotation request) is therefore CA-scoped and doesn't cover this.

Net: for the very common case of managed node auto-upgrade (AKS/EKS/GKE), there is no supported way to let a busy runner finish its job before its node is drained.

What I'd like

A supported mechanism to protect busy runners from voluntary node disruption. Options, roughly in order of preference:

  1. Make PDBs work — give EphemeralRunner a scale subresource (reporting 1/1), or reparent runner pods to a set-level owner (EphemeralRunnerSet) that has one, so a standard PodDisruptionBudget can protect runner pods. This is the cleanest fit with how drains already work. (Reconsider #4493 with the node-drain use case, not just the API-compat framing.)
  2. Native busy marker + graceful drain — controller sets a configurable label/annotation when a job lands and clears it on completion (#2562), and on receiving SIGTERM from a drain the runner finishes (or re-queues) the current job within terminationGracePeriodSeconds instead of dying immediately.
  3. At minimum, document that ARC v2 runners cannot be protected from node drains and that the only mitigation is at the infrastructure layer (maintenance windows / controlled upgrade cadence).

Related: #4493 (PDB scale subresource, closed won't-fix), #2562 (busy annotations for CA), #4148 (EphemeralRunner stuck Running after node drain).

Environment
  • gha-runner-scale-set / gha-runner-scale-set-controller 0.14.2
  • AKS, node auto-upgrade (drain via eviction API)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the EphemeralRunner CRD's status-only subresource and the controller behavior during node-drain termination; compare the findings with #4493, #2562, and #4148. Reproduce the AKS-style eviction scenario against controller 0.14.2 or master. Done requires an agreed, supported protection mechanism for busy runners, or explicit documentation of the infrastructure-only mitigation.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go, kubernetes
Domain
ci-cd, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.