AutoscalingRunnerSet stuck permanently "Outdated" when template.spec references a private container image (change-hash teardown never requeues)

AutoscalingRunnerSet stuck permanently Outdated when template.spec references a private container image (change-hash teardown never requeues)

Filing against / relating to #4596. This is a minimal, field-isolated reproduction: with an otherwise byte-identical template, a public runner image is stable and a private image (same registry host, same auth available) drives a permanent Outdated state with no listener and no requeue.

Environment

Chart	gha-runner-scale-set 0.14.2 (latest)
Controller	gha-runner-scale-set-controller 0.14.2 (app.kubernetes.io/version=0.14.2)
Install method	Helm, OCI: oci://ghcr.io/actions/actions-runner-controller-charts/
Config	org-level (githubConfigUrl: https://github.com/<org>), PAT auth, dedicated runner group
Kubernetes	v1.35.0 (Talos v1.12.7, containerd 2.1.7), Cilium CNI 1.16 (egress to world:443 from arc-systems confirmed allowed)
Registry	private image on ghcr.io, same host as the (public) default image

Summary

Replacing only the image reference in template.spec.containers[runner].image from a public image to a private image (on the same ghcr.io host, with a valid pull secret and a PAT that has read:packages — the package metadata returns HTTP 200) causes the AutoscalingRunnerSet to enter status.phase: Outdated permanently:

• the actions.github.com/runner-scale-set-id annotation is stripped,
• the listener pod is deleted and never recreated,
• the scale set registration is deleted from the Actions service,
• the controller returns from the reconcile without requeuing, so it never re-registers.

The controller loops (or goes idle) on:

Ephemeral runner set is outdated ...
  → Deleting the listener
  → deleting runner scale set
  → Deleted runner scale set from Actions service

This matches the mechanism described in #4596 (integrity/change-hash mismatch → teardown → no requeue), but the trigger here is specifically a private image reference — not a broad template change. Every other field of a full custom template is stable (see the isolation matrix below).

Reproduction

Start from a working default install (no template: block → stable). Then apply a template whose spec.containers[0].image is the only meaningful change.

Stable (public image):

template:
  spec:
    # ... full custom affinity / resources / volumes / securityContext ...
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest   # PUBLIC
        command: ["/home/runner/run.sh"]

Broken (private image) — same file, only the image line differs:

        image: ghcr.io/<org>/<private-runner>:latest    # PRIVATE (also fails by digest)
        # imagePullSecrets + read:packages PAT both present; `docker pull` works out-of-band

Isolation matrix (all watches are sustained ~5 min, sampled every 20s — short polls false-positive because the set flaps briefly on apply)

Template	Result
No template: block (default)	✅ Stable — registers, listener Running, phase Running, holds
Full custom template, public default image	✅ Stable — id constant, listener Running, runners land per affinity
Full custom template, private image (by :latest tag)	❌ Permanent Outdated, no listener, no requeue
Full custom template, private image (by @sha256: digest)	❌ Permanent Outdated
Minimal template, private image + explicit imagePullPolicy	❌ Permanent Outdated

The only variable that flips stable↔broken is whether the referenced image is public or private. Affinity, resources, memory-backed emptyDir volumes, and securityContext are all stable on the public image.

Trace — stable (full custom template, public image)

t=20..300s (every 20s): id='23' phase='Running' listener='Running'
verdict: distinct ids: 23 (constant) | outdated events(5m): 0
runner pods: 3x Running, all placed per nodeAffinity

Trace — broken (same template, private image)

t=20..300s (every 20s): id='∅' phase='Outdated' listener='none' runnerpods=0
verdict: distinct ids: (none) | outdated events(6m): 12
final: phase=Outdated, no listener, no runner scale set registration

Ruled out (does not fix it)

• PAT scope — added read:packages; the private package's metadata endpoint returns HTTP 200; pull secret is valid; docker pull of the exact digest succeeds out-of-band. Still Outdated.
• Tag vs digest — both fail identically.
• imagePullPolicy — explicit IfNotPresent/Always makes no difference.
• Clean reinstall — helm uninstall + reinstall of both releases; controller restart; helm --force; CR delete/recreate. All still stick.
• The documented #4596 status-patch workaround (patch status.phase → Pending) — yields exactly one create, then re-sticks Outdated.
• Admission/mutating webhooks — none touch the CR; the stored ARS template is clean.
• Network policy — arc-systems egress to world:443 is allowed and verified.

Why this looks like a hashing bug, not an auth bug

If this were purely an image-pull/auth failure, we'd expect the scale set to register and then the runner pods to fail ImagePullBackOff — but the failure is upstream of that: the listener and the scale-set registration themselves are torn down and never recreated, so no runner pod is ever scheduled. That points at the change-hash/integrity comparison in the controller treating the private-image template as perpetually outdated and taking the teardown-without-requeue path.

Confirming the image is genuinely pullable: both the tag'd and digest'd private images are already present in the node's image cache (crictl images / node status.images shows ghcr.io/<org>/<private-runner>@sha256:… resident). So the controller is rejecting a template whose image the cluster can and does pull successfully.

Ask

1. Is the change-hash / template-integrity computation sensitive to something about a private image reference (e.g. an image string it can't resolve, or a pull-secret-derived field) in a way that makes it never converge?
2. The teardown path returning without requeue is the core severity here — even a transient mismatch becomes permanent. Should that path requeue?

Happy to provide full controller logs, the exact (redacted) values, and CR YAML on request.

Start by tracing the controller's change-hash or template-integrity comparison and the Outdated teardown path described in the issue. Reproduce the public/private image matrix, then verify that a private image no longer causes permanent Outdated status and that the listener and scale-set registration recover through requeueing.