Registration token endpoint returns 502 in bursts; ephemeral runners die before registering

What

POST https://api.github.com/repos/{owner}/{repo}/actions/runners/registration-token returns 502 Bad Gateway under bursts of concurrent requests, with no clear retry hint. Self-hosted runners spawned from an autoscaler (in our case Modal-hosted ephemeral GHA runners) fail to register and exit immediately.

Observed 2026-05-04 ~16:00–17:00 UTC. ~21 ephemeral containers died mid-registration during a webhook-driven spawn burst, requiring manual cleanup of phantom Modal containers.

Repro

1. Configure many ephemeral runners to register on workflow_job.queued webhook (we do ~20 concurrent at peak).
2. Observe the runner-side stack trace from _mint_registration_token:

File "/root/app.py", line 247, in _mint_registration_token
    with urllib.request.urlopen(req, timeout=15) as resp:
  ...
urllib.error.HTTPError: HTTP Error 502: Bad Gateway

3. The container's runner.sh fails before config.sh registers; the slot is wasted.

What's missing on the runner side

The reference runner script _mint_registration_token flow doesn't retry 5xx — a single transient 502 from api.github.com kills the registration. For ephemeral spawns this is particularly painful because each failed registration burns a Modal container slot.

Suggested fix (runner-side)

Add bounded retry-with-backoff on 5xx responses to the registration-token call in the example runner setup script and document the pattern. Something like:

for attempt in range(4):
    try:
        with urllib.request.urlopen(req, timeout=15) as resp:
            return json.load(resp)
    except urllib.error.HTTPError as e:
        if 500 <= e.code < 600 and attempt < 3:
            time.sleep(2 ** attempt)
            continue
        raise

The github.com side may also want to investigate the 502 burst pattern, but a retry on the runner side dramatically reduces user-visible impact regardless.

Impact

Empty merge queue for ~3 hours despite ample Modal A100 quota; the 20-cap on max_containers was being held by phantom containers that 502'd during registration but Modal still counted their slot.

Locate the example runner setup script and the _mint_registration_token flow; inspect how runner.sh invokes registration before config.sh. Reproduce the registration failure under concurrent requests, then verify that bounded 5xx retry behavior is covered and documented without changing the GitHub API side.