calibration_diagnostics.json has no declared schema and is not validated on write

calibration_diagnostics.json is a required release artifact with no declared schema. Its shape is defined twice, in two libraries, in two different styles — implicitly by construction in the writer, explicitly by inspection in the contract — and nothing validates it at the point of writing. A second producer then diverged, which is the visible symptom rather than the defect.

There is no schema object

packages/populace-calibrate/src/populace/calibrate/diagnostics.py (392 lines) declares no class, no dataclass, no TypedDict, and uses no schema library. It is ten module-level helpers each returning dict[str, object], composed by diagnostics_payload(), whose signature returns a bare -> dict.

Nothing is validated on write

write_calibration_diagnostics (diagnostics.py:359) performs exactly one check — allow_nan=False, which rejects non-finite floats and nothing else — then serializes.

schema_version: 5 is written as a literal into the payload dict. The constant is a label the writer stamps, not a version of anything the writer enforces: raise CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION to 6 and the writer emits 6 while producing byte-identical output. A field could be dropped, renamed, or change type and the write would still succeed.

The real schema lives on the read side, in another library

_check_calibration_diagnostics (packages/populace-data/src/populace/data/contract.py:623) is where the structure is actually asserted: six required top-level sections with expected types, eleven required fields on every target row, non-empty source, measure as a mapping, and a cross-check that target_surface.n_targets == len(targets).

That is a real schema. It is just in the wrong place and the wrong form:

• Wrong form. Imperative checks appending strings to a failures list. Nothing can be introspected, reused by a producer, or published for consumers.
• Wrong time. It runs at populace-publish-release, potentially hours after the build wrote the file — so the feedback arrives at the end of a two-hour build rather than at the moment the payload is constructed.
• Wrong library. The writer is in populace-calibrate; the validator is in populace-data, which cannot import populace-calibrate (the dependency runs the other way). That is precisely why CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION is declared twice — diagnostics.py:43 and contract.py:98 — and held equal by test_calibration_diagnostics_schema_lockstep (packages/populace-build/tests/test_us_fiscal_refresh_builder.py:9539) rather than shared.

A constant duplicated across a package boundary and reconciled by a test is a workaround for a missing shared definition.

The symptom: a second producer that diverged

The ACS local-area build hand-assembles the same filename at tools/build_us_acs_local_release.py:1033, while the national build calls the library writer at tools/build_us_fiscal_refresh_release.py:6239. The two published artifacts already share no top-level key:

// releases/populace-us-2024-buildo-sparse-rmloss100-22bd902-20260722T232627Z/
{ "schema_version": 5, "weight_entity": "household", "options": { "method": "adam", … } }

// releases/populace-us-2024-buildo-acs-local-77e2061-20260724T110908Z/
{ "households": 1588854, "n_targets": 4461, "families": "snap,medicaid,soi",
  "geographies": "state,cd", "matrix_format": "sparse_csr", "matrix_shape": [4461, 1588854], … }

The local-area file carries no schema_version at all, and nothing catches that either: _check_calibration_diagnostics has exactly one call site (contract.py:1479), in the national branch of validate_release_dir. _validate_local_area_release_dir (contract.py:1036) never reaches it — zero occurrences in its body.

This divergence was possible because there was no single artifact saying what the file is. Adding a check for the second producer treats the symptom; a second producer could only ever have diverged.

Proposed direction

1. Declare the payload as types. A CalibrationDiagnostics dataclass (or TypedDict set) covering the top level plus TargetRow, TargetSurface, TargetRegistryRef, LossPoint, SkippedTarget. The eleven required target-row fields and six required sections currently enumerated in contract.py become the field list, in one place.
2. Validate at construction, not at publish. diagnostics_payload() returns the typed object; write_calibration_diagnostics serializes it and fails there if it does not satisfy its own schema. A build learns immediately instead of at the end.
3. Put the definition where both libraries can see it. The types belong somewhere populace-calibrate (producer) and populace-data (contract) can both import without inverting the dependency — either the kernel, or a small shared schema module. That retires the duplicated constant and its lockstep test in favour of one declaration.
4. Make the contract check a second line of defence. Keep validate_release_dir verifying the artifact on the way out — it guards against files produced by older code or edited by hand — but have it check against the shared schema rather than restating it.
5. Then converge the producers. With a schema that both can construct, the local-area build either satisfies it or declares a different artifact under a different filename. That decision becomes explicit instead of implicit.

Steps 1–3 are the fix; 4 preserves existing coverage; 5 is what closes the divergence.

Context

Found while auditing how the national and local-area builds diverge, during the staging-telemetry work in #563. Nothing in that PR touches this. Verified against main at 3537155; line numbers drift, so re-check before relying on one.

Read packages/populace-calibrate/src/populace/calibrate/diagnostics.py and _check_calibration_diagnostics in packages/populace-data/src/populace/data/contract.py, then inspect the two producer entry points at tools/build_us_acs_local_release.py and tools/build_us_fiscal_refresh_release.py. Use the existing contract fields and test_calibration_diagnostics_schema_lockstep as references. Done means both libraries share one declared schema, writes validate it, and producer divergence is explicit.