E2E visual regression: baselines are stale and a size mismatch leaves nothing to diagnose it with

Problem

The visual-regression suite is red on main, and when it fails it leaves nothing behind to diagnose it with.

Both remaining comparisons fail, in the Playwright container the baselines are meant to be compared in:

Snapshot observability-dashboard-page.png dimensions differ: expected 1280x3119, actual 1280x3127.
Snapshot admin-landing-page.png dimensions differ: expected 1280x5638, actual 1280x5198.

Failed!  - Failed:     2, Passed:     0, Skipped:     0, Total:     2

Reproduced identically on two consecutive runs, so the pages themselves are steady — the committed baselines are simply out of date. The last time they were refreshed was mid-July; the admin landing page has lost 440px of height since, and the observability dashboard has gained 8px.

Nothing noticed, because these comparisons are excluded from CI by filter and gated off behind CPD_E2E_VISUAL_REGRESSION. They only ever run when somebody asks for them by hand, so they can rot indefinitely and the next person to run make test-e2e-visual inherits the rot.

Investigating that turned up three separate things: one gap in the tooling, one fixable source of drift, and one page that cannot be baselined at all in its current form.

Evidence

1. A dimension mismatch is the one failure that writes no artefacts.

PageSnapshotExtensions.MatchSnapshotAsync writes an expected/actual/diff trio to Snapshots/diffs/ when the images diverge per-pixel, and the README tells you to go and look at them. On a size mismatch it throws with the two heights and writes nothing at all. That is the harder failure of the two to diagnose — "the page got 440px shorter" does not say which 440px — and it is the one with no evidence attached. The only way to see the images is to patch the helper and run again, which is what happened here.

2. The admin chrome carries live operational data into the compared region.

Both snapshots are admin pages, so both include the service-navigation Messages badge. Its number is the unread search-feedback count plus the dead-letter depth, and its colour keys off the dead-letter count alone (MessagesBadgeViewModel). Both halves move underneath the suite: a full E2E run took the local badge from 2264 to 2267.

Measured on /admin at the 1280x720 viewport the comparison uses, screenshotting the same page twice — once with the live count, once with the badge pinned:

differing pixels 1870 of 6653440 = 0.0281% (threshold 0.5000%)

Not a failure on its own, but a baseline that is guaranteed wrong from the moment it is taken, spending 6% of the pixel budget on a number no baseline can ever pin. The tag is also ~12px wider per digit, so a count that gains a digit moves everything after it along the navigation row.

3. The observability dashboard cannot be baselined against a live pipeline.

This one is not a stale baseline and it is not the badge. make test-e2e-visual runs the functional tests and the comparisons in the same pass, so by the time the dashboard is captured the suite has pushed requests through the pipeline and the worker has been running throughout. Refreshing the baseline and re-running the very next canonical pass still fails:

Snapshot observability-dashboard-page.png dimensions differ: expected 1280x3159, actual 1280x3127.
Failed!  - Failed:     1, Passed:   239, Skipped:     0, Total:   240

The diff shows why. Almost everything on the page is live telemetry: the three headline tiles (Processed today 10 → 20, Average end-to-end 4s → 2m, Current depths 0 → 6), the status sentence that quotes those numbers in prose, the health strip's colour and state, the per-queue waiting counts on the board, and all four charts, which are SVG rendered from the live series. The height moves as well, because the health strip's "Why?" reasons list grows and shrinks with the state — that is the 32px above.

StabiliseAsync already suppresses the board's "Recent transitions" list for exactly this reason. That was the first of these regions to be noticed; it was not the only one.

Proposed fix

For (1) and (2), which are straightforward:

1. Write the expected and actual PNGs on a dimension mismatch too, into the same Snapshots/diffs/ directory the pixel-divergence path already uses, and name the directory in the exception message. No overlay in this case: a per-pixel diff needs two images of the same size.
2. Pin the badge before a snapshot. Give StabiliseAsync an opt-in neutraliseLiveCounts flag that flattens the Messages tag to a grey zero, and pass it from the snapshot tests. Default off, so functional assertions and the review screenshots under Snapshots/search-ux/ still see the real number — the badge count is the subject of an assertion in MessagesGroupAndDetailTests.
3. Refresh the baselines from current main in the pinned container.

For (3) there is a decision to make rather than an obvious fix, and it wants its own ticket:

• Seed the dashboard from a fixed dataset for the snapshot — a pinned window of pipeline events so the tiles, charts and health state all resolve to the same values every time. Most faithful, most work.
• Pin the live regions the way the badge is pinned: fixed placeholder text in the tiles, a forced health band, charts suppressed. Cheap, but what is left to compare is mostly layout.
• Retire the comparison. The dashboard is a live board; a pixel baseline may simply be the wrong tool for it, and the functional observability tests already cover its behaviour.

Acceptance criteria

• A dimension mismatch writes {name}.expected.png and {name}.actual.png to Snapshots/diffs/, and the failure message points at the directory
• The Messages badge is pinned for the snapshot comparisons and left alone everywhere else
• Snapshots/linux-chromium/admin-landing-page.png matches what current main renders
• The admin landing comparison is green on consecutive canonical runs, including after a full E2E run has moved the badge count
• The E2E README describes the dimension-mismatch artefacts and when to pin live counts
• The observability dashboard's live-data problem is captured as its own ticket with one of the three options chosen

Notes

Worth deciding separately whether these comparisons should run anywhere automatic. As things stand they are a manual tool with no owner, which is how a two-month-old baseline goes unnoticed; the alternative is a scheduled or label-gated job that at least fails loudly.

Start with PageSnapshotExtensions.MatchSnapshotAsync and the existing Snapshots/diffs/ handling, then inspect StabiliseAsync, MessagesBadgeViewModel, and MessagesGroupAndDetailTests to understand the badge behavior. Check the E2E README and the admin baseline at Snapshots/linux-chromium/admin-landing-page.png. Done means dimension mismatches leave both PNGs, snapshot comparisons neutralise only the badge, baselines are refreshed, and the dashboard decision is captured separately.