Upstream-freshness dashboard: when was each dataset last checked against its source, and does it still match?

For every dataset with an upstream source, show when it was last checked against that source and whether it still matches — so upstream drift is something the repo notices, rather than something someone happens to discover while writing a manifest.

The audit dashboard already answers "where does the lecture series read its data from?". This answers the complementary question: "is the data we serve still what the source publishes?" Same generated-not-hand-tended principle.

Why now

The mpd2020.xlsx delta (#39) was found by accident. Nobody was looking for it; it surfaced because writing a manifest happened to involve re-downloading the file, and comparing it took a few minutes. Three header labels had been sitting divergent from upstream since 2023, invisible, and load-bearing for a published lecture.

That is the argument for automating the check. The information needed is already in the manifests — source, integrity.upstream.status, integrity.upstream.date — it is simply never revisited after the day it is written. A manifest's verified: 2026-08-03 is a claim with a shelf life, and nothing currently ages it.

What it would show

One row per dataset with an upstream source:

Column	Source
Dataset	manifest filename
Class	verbatim / constructed / dynamic-snapshot — determines how it is checked
Upstream	source.url, and whether a fetchable source.data_url exists
Last checked	integrity.upstream.date
Age of that check	derived — the number that should drive attention
Status	verified / spot-checked / unverifiable / drift detected
Result of the last automated check	matches / differs / not checkable

With the freshness treatment the audit overview already uses for its build age: green when recent, amber when stale, red when a check has actually failed. "Never checked" and "cannot be checked" must be visually distinct from "checked and fine" — the current catalog's ⚠️ unverifiable conflates unknown with unverifiable, and they are different.

Three check types, because one method does not fit

Grouping the 18 current manifests by what a check would actually mean:

Check	Applies to	Method
Re-fetch and diff	files with a direct upstream file URL	download, compare against integrity.sha256, and on mismatch diff the parsed content so the report says what changed rather than "hash differs"
Re-run the builder	constructed datasets with a committed builder	run it, compare the overlap window with the committed snapshot — this is the validate() stage of the four-stage contract (#14), reused
Not checkable	no source, no builder, or a prose-only provenance	show honestly as such. caron.npy and nom_balances.npy have no recorded source at all; they should read "cannot be checked", never "stale"

The blocker worth knowing before scoping this

Of the 18 manifests today, 16 have an upstream URL but only one has a directly fetchable file URL. The other fifteen point at a landing page, a search interface, or a data portal — https://ec.europa.eu/eurostat/data/database, https://earthquake.usgs.gov/earthquakes/search/, and so on. Those cannot be fetched and compared by a script.

So the first step is not the dashboard, it is the manifests: add a source.data_url (the exact retrievable artifact) wherever one exists, distinct from source.url (the human landing page). mpd2020.xlsx.yml in #38 already carries both and is the model.

For sources with no stable artifact URL — a query interface, an export builder — the honest answer is that the committed builder is the retrieval spec, and the check is "re-run the builder", not "re-fetch the file". Six of the seven verified datasets have committed builders, so that path covers most of what matters.

Current state, for scoping:

	Count
Manifests	18
Have any upstream URL	16
Have a fetchable data_url	1
Have a committed builder (re-runnable)	6
unverifiable — no automated check possible	11
No recorded source at all	2

Where it should live

Extend the existing dashboard rather than building a second one. scripts/build_audit.py already loads every manifest and render_audit.py already has the freshness-badge treatment; this is a section, not a new site. The scheduled audit-dashboard workflow is the natural place to run the checks — weekly is right for this, and the drift-alarm inbox added in #29 already exists to make a failure land somewhere a human sees it.

One design caution: a failed upstream check must not fail the build the way a failed internal-consistency check does. Upstream moving is normal and is not a defect in this repo; it should open or update an issue, not block the Pages deploy. Keep the two failure classes separate, the same way #14 separates a ValidationError from an infrastructure failure.

Relationship to the other threads

• #39 is the human register of known deltas. This issue automates finding them; #39 remains where a delta is reasoned about and resolved. Ideally a detected drift files itself into #39.
• #14 defines the builder validate() stage that the "re-run the builder" check reuses. Worth building this after that lands, so there is one validation implementation and not two.
• Phase 5's sources-alive canary is adjacent but narrower — it asks "is the source still reachable", this asks "does the source still agree". The canary is the right place to host the fetch machinery.

Part of #8. See PLAN Phase 5 and Phase 7.

Start with scripts/build_audit.py, render_audit.py, the 18 manifests, and the scheduled audit-dashboard workflow; compare the existing freshness-badge treatment with the upstream fields. Read #14's builder validate() contract and the Phase 5 sources-alive canary before choosing the check paths. Done means the existing dashboard reports freshness and drift distinctly, records uncheckable sources honestly, and upstream failures reach the drift-alarm inbox without blocking deployment.