test-containers-lectures: lecture-python.myst doubled to the 120-minute ceiling

lecture-python.myst in the weekly container-validation matrix roughly doubled from ~54 minutes to over 120 over two weeks, and now hits the job ceiling. #140 restores the alerting that should have surfaced this three weeks ago; this issue is the diagnosis and the capacity decision it exposed.

What the run history shows

Last green sweep was 2026-07-27. Durations for lecture-python.myst, both images, since:

The other two repos are stable and nowhere near the ceiling: lecture-python-intro 12–19m, lecture-python-advanced.myst 43–52m. So this is specific to one repo, not a runner-wide or image-wide slowdown.

The 08-10 leg finishing at 109m is the more alarming datapoint than the timeouts — it means the job now passes or fails on the margin, so the matrix will flap rather than fail cleanly.

What to work out

• Whether the cause is upstream content (lectures added or made heavier in lecture-python.myst), the container baseline, or execution behaviour — a git log over that repo's lectures/ across the window, against a manual dispatch at a pinned older image, separates the two
• Whether both images regressed by the same factor, which would point at content rather than the image
• Whether the GPU-dependent lectures are executing on CPU runners and dominating the time

The capacity decision this forces

timeout-minutes: 120 at .github/workflows/test-containers-lectures.yml:37 is a single ceiling shared by every leg, sized for the smallest. Options, in rough order of preference:

• Split the three builders into separate jobs, so html (where execution happens) is timed and reported independently of pdflatex/jupyter rendering — also makes the next regression legible without a stopwatch
• Raise the ceiling for the known-heavy repos only, per matrix entry rather than per workflow
• Reduce what this repo validates weekly, if the answer is that a full three-builder pass over the largest lecture repo is no longer the right weekly cost

Preventive half

Slow drift is invisible here by construction: the only symptom is duration, and nothing records it. #140 now reports per-leg duration inside the failure issue, which helps only once something already broke.

• Record per-leg durations to $GITHUB_STEP_SUMMARY on every run, and warn when a leg exceeds the previous successful run's time by a threshold

Comparing against the previous successful run (via the runs API) rather than a checked-in baseline is deliberate: these durations track upstream lecture content this repo does not control, so a committed baseline would go stale on every upstream lecture addition and become noise nobody maintains — which is how containers/VALIDATION.md's timing table died.

This is the third silent-duration failure in this repo, after #103 (nine weeks of cold-but-green container builds) and #83 (two months of 40-minute cold lecture builds). In all three the only observable symptom was time.

Start with .github/workflows/test-containers-lectures.yml:37 and inspect the weekly run history, then compare a git log of lecture-python.myst/lectures with a manual dispatch using a pinned older image. Review the runs API approach and containers/VALIDATION.md before determining the capacity and duration-reporting changes; done requires an agreed implementation scope and acceptance criteria.