Task-runner toolchain drift: nothing refreshes the image on the target repo's clock

The task-runner image freezes the target repo's toolchain, and nothing refreshes it on that repo's clock. This has now broken the nightly integration-failure triage three times:

| Date | Drift | Cost |
| --- | --- | --- |
| 2026-08-17 | `tokenizers` 0.22.2 vs main's new `>=0.23.1` | 3 groups, ~3.5M input tokens ([transformers#48037](https://github.com/huggingface/transformers/issues/48037)) |
| 2026-08-18 | `transformers-mlinter` 0.1.2 vs the 0.1.4 rule schema | 1 checker dead |
| 2026-09-12 → 09-13 | `huggingface-hub` `[1.5, 1.31)` vs main's new `>=1.31.0` ([transformers#48685](https://github.com/huggingface/transformers/commit/80a79d1b)) | 5 groups over two nightlies, ~8.4M input tokens ([transformers#48749](https://github.com/huggingface/transformers/issues/48749), [#48778](https://github.com/huggingface/transformers/issues/48778)) |

#125 makes the failure cheap and loud — a ~10s preflight instead of a full agent loop per candidate. It does **not** shorten the window in which prod is broken. That is what this issue is for.

## Why the window exists

1. **`docker.yml` only fires on a push to serge `main`.** The Dockerfile re-resolves `transformers[quality]` from `main` at build time (#89), so a rebuild is *sufficient* — but nothing makes one happen. serge's last push was 2026-09-09; transformers moves its pins whenever it likes.
2. **You cannot refresh the toolchain without a code commit.** `update-deploy.md` §0 forbids re-dispatching `docker.yml` onto a sha prod already runs, because it would overwrite the tag prod is running with different contents. So "the toolchain is stale" currently requires inventing a commit.
3. **A rebuild alone changes nothing in prod.** `taskExecution.kubernetes.image` is a hand-pinned `sha-` in the deployment values; it needs an edit and a helm upgrade.
4. **Nothing watches.** The broken gate surfaces as `⚠️ task failed` in a tracking issue generated at 22:30 UTC. Two nights passed before anyone looked.

The root design problem is in (1)+(3): the task-runner image mixes **two axes that move independently** — serge code (moves with serge commits) and the transformers toolchain (moves with transformers) — behind a single `sha-` tag. A tag that names only one of the two inputs cannot express "same serge, fresher toolchain".

## Options

**A. Split the tag.** Give the task-runner a `-` (or a plain dated) tag and let a scheduled build publish a fresh toolchain against the deployed serge sha. Preserves tag immutability, makes (2) go away, still needs a pin bump for (3).

**B. Scheduled rebuild + auto-bump.** A daily workflow (before 22:00 UTC, after the base image's `6 0 * * *` build) rebuilds and opens a PR against the deployment values with the new tag. Bounded window (~24h) and a human still approves what prod runs.

**C. Scheduled drift *detection* only.** Compare the deployed image's installed versions against transformers `main`'s `_deps` daily and alert. Cheapest; leaves the fix manual, which is where we already are.

**D. Allow PyPI through the egress allowlist** so the gate can drop `--no-deps` and resolve pins itself. Kills the class outright, and is the only option that does. It also widens the network surface of a pod running LLM-authored code — that is a real trade-off and probably a no, but it should be an explicit no rather than an unexamined one.

Note for whichever we pick: `docker.yml` uses `cache-from: type=gha` and the `RUN pip install --upgrade "transformers[quality] @ git+…@main"` string never changes, so on a day the base image digest does not move, buildx replays that layer and the re-resolve is a silent no-op. Any scheduled rebuild needs a cache-bust build arg to be trustworthy.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Start with docker.yml and the Dockerfile to understand image rebuilds and cache behavior, then read update-deploy.md §0 and the deployment values containing taskExecution.kubernetes.image. Compare the proposed tag, scheduled rebuild, drift-detection, and egress approaches, including their production-safety trade-offs. Done means an agreed design reduces toolchain drift without violating tag or deployment constraints.

Written by the indexing model from the issue text.