CI measurements v2: typed scenarios, truthful completeness, and cache-reuse proofs

Problem

The current CI measurement system can produce a green report whose evidence is incomplete and whose summary overstates what was proven.

PR #922 exposed the architectural gap:

• its final pre-rebase measurement artifact was partial, advisory, and enforceable=false;
• only 17 of 26 enabled observations were gateable and 10 baselines were missing;
• the report still summarized the result as “No regressions”;
• forced check:quick changed from 15.520 s to 17.884 s (+15.2%) while the aggregate label said “no material impact”;
• the scenario that actually changed—second-root materialization against a shared content-addressed store—was not modeled directly;
• flat duration/closure probes could not distinguish lifecycle state, cache reuse, contention, correctness, or incomplete evidence.

The post-rebase investigation found that 95% of the forced regression belonged to pnpm:install: paired aggregate deltas were +2272/+2399/+2451 ms, while the pnpm child contributed +2260/+2280/+2360 ms. The cause was one extra full health traversal (2 calls → 3), not Nix evaluation, Genie execution, lease waiting, or capacity admission. Warm cached pnpm increased only +79–85 ms, and the richer projection digest increased by about 124 ms paired.

This is exactly the kind of distinction the measurement system should make mechanically rather than through one-off archaeology.

Related evidence:

• #922
• green exact-head CI after conflict/FOD repair
• interactive benchmark and architecture preview

Outcome

Introduce a typed, versioned ci.measurement-run/v2 scenario envelope owned by effect-utils. Repositories declare scenarios, compatibility dimensions, correctness invariants, and budgets; the shared engine executes, validates, compares, reports, and exports them.

The system must answer four independent questions:

1. Compatibility: are baseline and candidate valid to compare?
2. Completeness: did every required observation produce usable evidence?
3. Correctness: did the scenario preserve its behavioral invariants?
4. Performance: did comparable measurements remain within their calibrated budgets?

A green performance verdict must never imply complete evidence or behavioral correctness unless those dimensions independently passed.

Requirements

R1 — Completeness is part of the verdict

• Every required scenario/observation has an explicit outcome: complete, missing, skipped, or failed, with a machine-readable reason.
• Required evidence that is missing, skipped, failed, or incompatible cannot summarize as “pass” or “no regressions”.
• Reports use precise language such as “No threshold violation among comparable observations; required evidence incomplete.”
• Failed paired samples remain first-class records; they are not silently dropped before aggregation.

R2 — Comparison identity is explicit and stable

• Each scenario declares a compatibilityDimensions whitelist.
• Only declared semantic compatibility fields participate in the comparison fingerprint.
• Repetition count, paired-order seed, run ID, timestamps, runner load evidence, and other execution metadata do not fragment baseline identity.
• Incompatible baseline/candidate pairs are visible evidence, not missing-data ambiguity.

R3 — Scenarios model lifecycle state

At minimum, scenarios can declare and record:

• store: empty or warm;
• materialization root: absent or present;
• task cache: cold or warm;
• network: online or offline;
• root relation: same root, second root/worktree, or concurrent roots;
• reset/setup receipt proving that the requested initial state was established.

R4 — Outcomes are multidimensional

The shared result type supports:

• total duration and named phases (resolve, import, project, verify, admission/lease/prune waits);
• reused and downloaded objects/bytes;
• selected and observed materialization method (clone, hardlink, copy, other);
• logical bytes, physical bytes, unique blocks, and shared-block ratio where the platform can prove them;
• peak RSS, CPU time, and I/O counters where affordable;
• correctness invariants and their proof receipts;
• warnings and capability-qualified missing values.

Zero is never used to mean “not measured”. All categorized storage measurements reconcile to an explicit total plus unclassified.

R5 — Correctness is co-equal with speed

Scenarios can assert domain invariants such as:

• materialization remains root-local;
• virtual stores remain distinct between roots;
• ordinary packages reuse shared content while mutable/native leaves remain isolated as designed;
• second-root offline materialization succeeds without downloads;
• projection health/digest checks pass;
• concurrent roots complete without cross-root mutation.

R6 — Workload provenance is recorded

Every run records enough provenance to explain scale and reproduce the scenario: workspace/package/file counts, lock digest, relevant pnpm/devenv/protocol versions, OS/architecture, filesystem/device capabilities, and scenario implementation version.

R7 — Merge policy is separate from execution topology

• Branch protection depends on stable semantic gate names, not low-level job or matrix names.
• Existing required check names remain stable unless a coordinated workflow + generated repo-settings cutover is explicitly planned.
• Job topology, runner assignment, repetitions, and exporters can evolve without redefining what the merge gate means.

Proposed data shape

type MeasurementRunV2 = {
  readonly schema: "ci.measurement-run/v2"
  readonly scenario: {
    readonly id: string
    readonly version: number
    readonly compatibilityDimensions: ReadonlyArray<string>
  }
  readonly compatibility: Record<string, string | number | boolean>
  readonly execution: {
    readonly runId: string
    readonly runner: RunnerEvidence
    readonly pairedOrderSeed?: string
    readonly repetition: number
  }
  readonly stateBefore: LifecycleState
  readonly stateResetReceipt: ProofReceipt
  readonly outcome: {
    readonly lifecycle: "complete" | "missing" | "skipped" | "failed"
    readonly reason?: MissingReason
    readonly duration: DurationOutcome
    readonly phases: ReadonlyArray<PhaseOutcome>
    readonly reuse: ReuseOutcome
    readonly storage: StorageOutcome
    readonly resources: ResourceOutcome
    readonly invariants: ReadonlyArray<InvariantOutcome>
  }
}

The exact TypeScript representation may change, but the separations above are architectural invariants.

Initial scenario set

Use #922 as the first vertical slice:

1. warm store + same materialization root + warm task cache;
2. warm shared store + absent second worktree root + online materialization;
3. warm shared store + absent second worktree root + offline materialization;
4. two concurrent absent roots sharing the same store;
5. cold empty-store reference scenario on scheduled main.

Each scenario records phase durations, downloads/reuse, materialization behavior, storage evidence, and the root-local/distinct-store/offline-health invariants.

CI topology

Pull requests: required

• Fast deterministic topology and correctness scenarios.
• Evidence-completeness contract and schema validation.
• Stable semantic aggregate gate.

Pull requests: advisory during calibration

• Paired Linux warm same-root and second-root latency.
• Budget evaluation only after A/A noise and runner-class distributions are measured.

Scheduled main

• Cold empty-store materialization.
• Concurrent roots and lock/lease contention.
• Offline rematerialization.
• Physical/unique-block storage measurements.
• Higher-sample resource measurements and optional traces.

Darwin

• Keep a small PR-time topology/offline correctness smoke.
• Run cold, concurrent, storage-heavy, and high-sample scenarios on scheduled lanes because Darwin is scarce capacity.

Ownership

• effect-utils: schema, validator, scenario engine, comparison semantics, aggregation, reporter, capability adapters, and exporter interfaces.
• consumer repositories: scenario declarations, workloads, correctness invariants, and calibrated budgets.
• exporters / OTEL: diagnostic and trend attachments only; they are not merge authority.

Do not create a fake fleet-wide mandate. Prove the shared engine in effect-utils and one simple plus one complex consumer before staged adoption.

Delivery plan

Phase 1 — Truthful verdicts on the existing engine

• Represent completeness independently from threshold status.
• Replace “No regressions” when evidence is partial.
• Preserve failed/incompatible samples in artifacts and reports.
• Add reconciliation/unclassified handling for storage categories.

Phase 2 — Typed v2 core

• Add the versioned schema, validator, stable compatibility fingerprint, lifecycle state, proof receipts, and invariant outcomes.
• Add schema fixtures and property/round-trip tests.
• Provide a v1 reader or explicit clean-cut migration; do not maintain two competing write authorities.

Phase 3 — #922 vertical slice

• Implement the five initial scenarios.
• Produce phase/reuse/storage/correctness outcomes from one execution rather than overlapping shell traversals.
• Calibrate paired Linux noise with A/A runs before setting warning/failure budgets.

Phase 4 — Semantic gates and scheduled lanes

• Keep required check names stable while replacing their internals.
• Add scheduled Linux/Darwin long-tail scenarios.
• Update generated workflow and repo-settings sources together if any gate migration is unavoidable.

Phase 5 — Cross-repo adoption

• Integrate one simpler and one more complex sibling repository.
• Document the extension contract and repo-owned scenario/budget boundary.
• Add exporter support only after the authoritative local artifact and verdict are stable.

Acceptance criteria

• A fixture with 1 missing required observation cannot render pass or “No regressions”.
• Baseline and candidate with identical declared compatibility but different seeds/run IDs compare successfully.
• Changing a declared compatibility dimension makes the pair explicitly incompatible.
• Failed paired samples remain visible and affect completeness.
• The second-worktree online scenario proves zero-download reuse when the store is warm.
• The second-worktree offline scenario succeeds and proves root-local health.
• The concurrent-root scenario proves distinct virtual stores and no cross-root mutation.
• Storage categories reconcile to total + unclassified; unsupported measurements are absent-with-reason, never zero.
• Reports present compatibility, completeness, correctness, and performance as separate statuses.
• A/A calibration evidence and the resulting budget rationale are checked into the policy/spec surface.
• PR and scheduled topology are generated from source-of-truth .genie.ts files.
• Required semantic check names remain stable, or the coordinated repo-settings cutover is proven.
• One simple and one complex consumer demonstrate repo-owned scenarios using the shared effect-utils engine.

Non-goals

• Making every expensive measurement required on every PR.
• Treating serialized NAR closure size as physical disk usage.
• Treating traces, dashboards, or an external benchmark service as merge authority.
• Encoding scenario catalogs, sample counts, job topology, or dashboard presentation in constitutional requirements.
• Solving noise by widening thresholds before A/A calibration.
• Adding more flat probes without the typed lifecycle/correctness/completeness model.

Decision summary

The preferred architecture is the typed shared scenario envelope. A small patch that only improves labels and adds #922-specific probes is useful as Phase 1, but is not the end state. Trace-first instrumentation and an external benchmark service can attach diagnostics and trends later; neither should own correctness or merge decisions.

Posted on behalf of @schickling

field	value
agent_name	🦜 co3-macaw
agent_session_id	2704af84-dabe-4279-99fb-e8e7c3920c7a
agent_tool	Codex CLI
agent_tool_version	0.144.1
agent_runtime	Codex CLI 0.144.1
agent_model	unknown
runtime_profile	/nix/store/qq3avrif77r90ypqbdv3hgd3gvwj5s32-coding-agent-runtime-profile/share/coding-agents/profile.json
skills_manifest	/nix/store/pjlb3cwf453ghzmc8jj4v8h29sw6p4dg-agent-skills-corpus/share/agent-skills/manifest.json
worktree	dotfiles/schickling-assistant/2026-07-16-vista-tscheck-fix
machine	dev3
tooling_profile	dotfiles@4b8e1c5

Start by locating the source-of-truth .genie.ts files and the existing CI measurement engine. Read the Phase 1 requirements and the proposed MeasurementRunV2 shape, then inspect the schema fixtures and property/round-trip tests mentioned in the delivery plan. Done means completeness, compatibility, correctness, and performance are reported separately, with the listed acceptance criteria covered.