Epic: publish and discover exact-k dataset families through Logbook and immutable Hugging Face releases

Summary

Complete the exact-k direction in #578 with a publication and discovery layer that treats one authenticated pre-calibration pool as a dataset family containing its full and sparse variants.

This epic should build on the Logbook introduced in #632 rather than create a second mutable family registry on Hugging Face:

• Hugging Face stores immutable tagged family manifests, derivation bundles, child manifests, and dataset artifacts.
• Logbook provides the append-only discovery, relationship, and lifecycle index connecting build attempts, families, and variants.

After a family is published, an authorized maintainer should be able to:

1. discover the family and its existing variants through Logbook;
2. pull the family's immutable derivation bundle from Hugging Face;
3. build a new exact-k variant locally with one documented command;
4. validate it against the family's frozen targets and gate battery;
5. publish it to a family-scoped path at an immutable Hugging Face tag without rebuilding or replacing the full family; and
6. register it as another child of the same family in Logbook without moving the production default.

This epic is about family publication, reproducible derivation, discovery, and governance. It builds on #578's pool and exact-k machinery and #632's Logbook rather than replacing either.

Current capabilities and gaps

The repository now has important parts of the lineage:

• a simulation-ready pre-calibration pool and authenticated pool manifest;
• exact-k selection and fixed-support refit;
• child manifests that retain parent-pool provenance;
• a tag-only exact-k candidate publication lane; and
• the append-only Logbook build trace from #632, with a Supabase live store, public projection, and hash-chained git archive.

However, there is no first-class family object that associates all variants derived from the same pool. In particular:

• the calibrated k=N release is currently a sibling of sparse releases, but the siblings are not discoverable as one family;
• Logbook records neither requested/realized k nor family membership (#641);
• the exact-k launcher/publication path does not yet emit a corresponding Logbook record;
• no public relationship model connects a build attempt to a family, an immutable HF release, or sibling variants;
• the loader cannot enumerate or select k variants dynamically;
• one launcher invocation builds one of the currently ratified k values;
• L0 probabilities are not persisted for reuse across independently built variants; and
• there is no supported pull → build → validate → publish → register workflow for adding a child later.

Downloading only the finished k=N H5 is not enough to derive a principled child. Exact-k selection/refit needs the authenticated pre-calibration pool, original importance weights, frozen targets, and either the L0 selection probabilities or everything needed to reproduce them.

Why this structure is beneficial

1. Amortizes the expensive build. Source acquisition, imputation, simulation, target materialization, and the L0 selection solve can be performed once per family. New k values pay primarily for selection, refit, gates, and packaging.

2. Makes lineage explicit and auditable. Every child binds to one parent pool, target register, selection state, code/config identity, and gate battery. Users can distinguish genuine variants of one build from unrelated datasets that happen to have similar names.

3. Implements the record-count model from #578. Dense and sparse stop being disconnected product categories. The full dataset is k=N and each smaller dataset is another measured point in the same family.

4. Allows incremental publication. A useful new k can be added after the original family build without rerunning the complete upstream pipeline or replacing existing children.

5. Improves discovery and comparison. Logbook can expose k, ESS, fit loss, geography coverage, gate status, size, seed, and immutable HF revision side by side without requiring consumers to scan tags or parse release names.

6. Keeps artifacts simple to distribute. Once Logbook resolves a variant to repo + tag + manifest path, Hugging Face supports downloading the selected file or tagged snapshot directly.

7. Separates publication from promotion. Adding a certified or candidate child need not change latest.json or the default loader result. Promotion remains a separate, explicit operation.

8. Avoids a mutable cross-release index on HF main. Append-only Logbook membership records avoid concurrent family_manifest.json edits, lost updates, and conflict with the existing tag-only publication doctrine.

9. Creates a reproducible maintainer workflow. An authorized contributor can start from immutable Hub inputs rather than an undocumented build-machine directory and publish a byte- and provenance-verifiable result.

10. Generalizes across countries and tiers. The same family contract can support public and gated/private country families, subject to an explicit distribution policy for each derivation bundle.

Architectural responsibilities

Logbook discovery and audit plane
  family identity, variant membership, k, status, build row
                              │
                              ▼
Hugging Face artifact plane
  immutable tag, manifests, derivation inputs, H5, diagnostics

The two systems have distinct authority:

• An immutable HF manifest is authoritative for artifact bytes, hashes, package compatibility, and cryptographic lineage.
• Logbook is authoritative for public discovery, family membership, build-attempt linkage, and lifecycle history.
• The loader must cross-check the selected Logbook record against the immutable child manifest rather than trusting either representation alone.

Logbook should contain metadata and locators, not dataset payloads.

Proposed lineage model

The cryptographic root of a family should be the authenticated pre-calibration pool, not the finished calibrated k=N H5.

authenticated dataset family / pre-calibration pool
├── full variant: k=N
├── sparse variant: k=57,240
├── sparse variant: k=20,000
└── later independently built exact-k variants

The k=N release remains the human-facing “full build,” but technically it is one child of the pool alongside the sparse variants. This preserves the original-weight and Horvitz–Thompson refit contract.

A family ID should be content-addressed by, or at least bind, the parent pool-manifest digest. All children share that family ID. A derived_from_variant or support_subset_of relationship should only be recorded when it is actually true; independent exact-k draws are siblings, not necessarily nested descendants.

Illustrative immutable Hugging Face layout

Exact names are open to refinement. The important contract is that each locator includes an immutable HF revision.

At the immutable family-root tag:

families/{family-id}/
  family_manifest.json
  derivation/
    pool.h5
    pool.manifest.json
    target_register.json
    selection_state.npz
    build_spec.json

At each immutable child tag:

families/{family-id}/variants/{variant-id}/
  microcosm_{country}_{year}.h5
  build_manifest.json
  release_manifest.json
  calibration_diagnostics.json
  ...

The immutable family manifest identifies the derivation root and family contract; it is not a mutable registry that must be rewritten for every future child. Each child manifest points back to the family-root revision and derivation artifacts by path and SHA-256. Logbook supplies the evolving append-only membership index.

Existing release paths may remain available during migration. Every authoritative artifact reference must name an immutable revision and digest rather than rely on a mutable root copy.

1. Versioned family, child, and Logbook contracts

Add an immutable versioned family manifest containing at least:

• family ID, country, year, and source tier;
• parent pool release ID, path, SHA-256, and immutable Hub revision;
• frozen target-register, loss-basis, and gate-battery identities;
• selection-state/profile identity;
• canonical k=N child, if one exists at family creation; and
• schema and package compatibility versions.

Each child release manifest must independently bind:

• variant ID and family ID;
• immutable family-manifest revision and digest;
• parent pool identity;
• target-register and gate-battery identities;
• selection-state digest;
• requested k, realized k, record unit, and seed;
• exact support digest and inclusion-probability/refit receipts;
• build/config/package versions;
• artifact paths and hashes; and
• gate/certification evidence.

Extend Logbook with an append-only, publicly queryable family/variant read model. Preserve the existing build-row chain; use versioned row fields or insert-only extension/event tables keyed to authenticated build rows. The public model should contain at least:

• family ID and variant ID;
• originating build ID or Logbook row digest;
• requested k, realized k, and record unit (#641);
• seed and variant role (full or sparse);
• HF repository, immutable revision/tag, manifest path, and artifact digest;
• publication/certification lifecycle events; and
• optional typed relationships such as supersedes or support_subset_of.

Do not require consumers to parse build_id, release tags, artifact_location, or private/free-form gate_verdicts to discover these properties.

2. Distributable derivation bundle

Publish everything needed to derive a new child without access to the original build machine:

• simulation-ready pre-calibration pool;
• authenticated pool manifest and diagnostics;
• original importance weights;
• stable record identities;
• frozen target register and compiled-target identity;
• loss basis and gate-battery identity;
• calibration and refit configuration;
• Microcosm commit and PolicyEngine model/Core versions; and
• persisted selection state.

The preferred selection artifact is a versioned selection_state.npz containing per-record π aligned by stable record identity, plus the pool, target, solver-config, and code hashes governing those probabilities.

This requires splitting the current exact-k operation into independently callable stages:

1. learn and persist selection state;
2. select exact-k support from authenticated state;
3. refit and run the family battery; and
4. package a child release.

A per-country/tier policy must declare whether the derivation bundle is public, gated, private, or unavailable. Publication must never widen access to licensed microdata.

3. Local pull/build workflow

Provide documented commands along these lines:

microcosm family pull policyengine/populace-us \
  --family {family-id} \
  --with-derivation

microcosm variant build \
  --family ./{family-id} \
  --k 50000 \
  --seed 17 \
  --out ./variant-k50000

microcosm variant publish ./variant-k50000 \
  --repo-id policyengine/populace-us

The family lookup may begin in Logbook, but the pull command must:

• resolve an immutable family-root HF revision;
• selectively download the required derivation artifacts;
• verify all hashes and lineage bindings;
• enforce package compatibility; and
• report disk, memory, and expected runtime requirements.

The build command must:

• accept any feasible positive k, rather than only the initial ratified ladder points;
• reuse authenticated selection state;
• perform exact-cardinality selection;
• use the inclusion-aware w/q refit baseline;
• run the same frozen register and complete invariant battery;
• assert requested k equals realized household count; and
• write a self-contained child bundle without publishing it.

Build and publish should remain separate operations.

A child identity must bind at least family + k + seed + selection profile + refit configuration so multiple experiments at the same k cannot collide.

4. Immutable child publication and Logbook registration

Extend the immutable-first publisher with a family-child mode that:

1. revalidates the complete child bundle locally;
2. authenticates its parent family and selection-state identities;
3. refuses incomplete or failed required gate evidence;
4. uploads the family-scoped child folder to a temporary HF branch;
5. creates an immutable child tag and captures its commit revision;
6. refuses to overwrite an existing child path or tag;
7. durably spools an append-only Logbook family-membership/publication event containing the immutable HF locator and digests;
8. best-effort delivers that event to the Logbook live store, retaining it for reconciliation if delivery fails; and
9. leaves latest.json, HF main, and mutable canonical artifacts unchanged.

Hugging Face and Supabase cannot participate in one atomic transaction. The cross-service contract should therefore be immutable, idempotent, and reconcilable:

• If HF publication succeeds but live Logbook insertion is delayed, the tagged release remains valid but temporarily undiscoverable through Logbook; the locally spooled event can be retried.
• An identical already-published tag or Logbook event is a no-op.
• An identity collision with different bytes fails closed.
• A public Logbook relation whose HF manifest is absent or disagrees on identity/digest is excluded or marked invalid.
• The immutable child manifest contains enough evidence to reconstruct a missing Logbook registration under an authorized reconciliation process.

Publishing/registering a family member and promoting a variant to the default must be separate commands, permissions, and audit events. Certification, deprecation, and supersession should be append-only Logbook events rather than destructive updates.

5. Discovery and loading

Add family-aware APIs, for example:

microcosm.data.families("us", 2024)
microcosm.data.variants("us", 2024, family="{family-id}")
microcosm.data.load("us", 2024, family="{family-id}", k=50_000)
microcosm.data.load_variant("{variant-id}")

The discovery flow should:

1. query the public Logbook model for families or variants;
2. select a variant by family, k, certification status, seed, or explicit variant ID;
3. resolve its immutable HF repo + revision + manifest path;
4. download the child manifest at that revision;
5. cross-check its family ID, k, build identity, revision, and artifact digest against Logbook; and
6. download and SHA-verify the selected artifact and enforce package compatibility.

load_variant() must also accept an explicit release/tag so a known artifact remains downloadable if Logbook is temporarily unavailable. A frozen Logbook export may support offline enumeration.

The current low-friction default load should remain backward compatible. Family children should not require one committed package-registry entry per k, scanning every Hub tag, or consulting a mutable index on HF main.

6. Certification, permissions, and governance

Specify:

• who may publish directly to the PolicyEngine Hub repository;
• who may register, certify, deprecate, or supersede variants in Logbook;
• a staging/fork-and-import path for contributors without write access;
• candidate, published, certified, deprecated, rejected, and invalid-registration semantics;
• whether any gate-passing k is publishable or whether some values require ratification;
• how multiple children at the same k are represented and selected;
• promotion criteria for becoming the default;
• retention/deprecation behavior; and
• whether sparse variants are required to be nested.

All sparse children are subsets of k=N, but independently sampled k values are not necessarily subsets of one another. If k=20,000 must be contained in k=57,240, that requires a persistent ranking or coordinated nested-sampling design beyond the current independent Sampford draws. Typed support_subset_of relationships must only be registered when verified.

7. Documentation and dataset-card presentation

Document:

• family, child, and Logbook relationship schemas;
• Hub folder/tag conventions;
• the complete discover/pull/build/validate/publish/register tutorial;
• direct download by immutable tag as a fallback;
• hardware, disk, and runtime expectations;
• public versus licensed derivation policies;
• candidate versus certified semantics;
• promotion and deprecation procedures;
• reconciliation after HF or Logbook delivery failures;
• reproducibility guarantees;
• the nesting or non-nesting contract; and
• troubleshooting for identity, hash, compatibility, relationship, and gate failures.

Generate a public family comparison table from Logbook plus authenticated child manifests, showing each child's k, seed, ESS, fit metrics, geography-level coverage, gate/certification status, artifact size, release ID, and immutable HF revision.

Proposed increments

1. Logbook cardinality integration — implement #641 and make the exact-k launcher/publication path emit requested/realized k.
2. Logbook family/variant relationships — append-only family, membership, HF locator, and lifecycle contracts with public read views.
3. Immutable family and child contracts — family-root manifest, child ancestry block, HF paths/tags, validators, and migration plan.
4. Derivation bundle and selection-state seam — persist/load π by stable identity and split selection from refit.
5. Local discover/pull/build CLI — Logbook discovery, immutable HF resolution, verification, arbitrary feasible k, and child packaging.
6. HF publisher plus Logbook registration — immutable child tag, durable local event, best-effort live registration, and idempotent reconciliation.
7. Discovery and loader support — enumerate families/children and load an explicitly selected certified child.
8. Documentation and dataset-card automation — end-to-end maintainer runbook, governance, licensing, and comparison presentation.

Acceptance criteria

• A family root is published at an immutable HF revision with an authenticated derivation bundle and optional k=N child.
• Logbook contains a public family entry bound to that immutable family-root manifest and pool identity.
• A clean machine can discover the family, pull its immutable HF derivation bundle, and verify the complete derivation closure.
• One documented command can build a new feasible k child without rerunning the upstream source/imputation pipeline.
• The child contains exactly k households and passes the family's frozen register and gate battery.
• The child manifest binds the family, pool, selection state, targets, requested/realized k, seed, support, refit, code, and package identities.
• An authorized maintainer can publish the child at a family-scoped path under an immutable HF tag without overwriting any sibling.
• Publication durably creates or spools an append-only Logbook membership event pointing to the immutable HF manifest and artifact digest.
• Adding a child changes neither HF main, latest.json, nor the default dataset.
• Consumers can enumerate variants by family and k through Logbook and load an explicit certified child from HF.
• The loader cross-checks Logbook metadata against the immutable child manifest before accepting an artifact.
• A known child remains directly downloadable by HF tag when Logbook is unavailable.
• Delayed Logbook delivery after successful HF publication is recoverable through idempotent reconciliation.
• Corrupt parent, selection, target, support, artifact, relationship, or gate evidence is rejected before registration or loading.
• Concurrent publishers cannot lose or overwrite family members; identity collisions with different bytes fail closed.
• Existing default releases, tags, and loaders remain compatible during migration.
• Country/tier licensing rules are enforced for derivation artifacts and Logbook visibility.

Non-goals

• Treating the calibrated k=N H5 alone as sufficient derivation input.
• Storing dataset payloads in Logbook.
• Maintaining an append-mutated family registry on HF main.
• Pretending HF publication and Logbook registration are one cross-service atomic transaction.
• Giving arbitrary users write access to PolicyEngine's Hugging Face repositories or Logbook.
• Automatically promoting every published child to the default.
• Assuming sparse children are nested unless a separate nested-sampling contract is adopted and verified.

Relates to #578, #632, and #641.

Start by reading the exact-k machinery from #578 and the Logbook implementation from #632, then trace the launcher and publication paths described in the issue. Map the family and child manifest contracts, immutable Hugging Face locators, and Logbook extension points before splitting the epic into implementable changes. Done means an authorized user can pull, build, validate, publish, and register a new family variant without replacing existing variants or the production default.