Incremental/delta batch evaluation (design gated on MIE change-signal answer)

Start with docs/MIE_INTEGRATION_QUESTIONS_2026-07-09.md and resolve Q6, then read docs/superpowers/specs/ and the batch engine context, including PR #250's inInitialPopulation signal. Phase 1 is done when an owner-approved design covers the API-based watermark/hash scheme, proposed eval-state DDL, invalidation matrix, and parity plan. Phase 2 requires backend-ts typecheck and tests, including golden parity and auditable evaluated-versus-skipped accounting.

Context

Recurring batch runs over ~100k should re-evaluate only subjects whose data changed. Deliberately NOT built yet: the right design depends on what change signals WebChart's API exposes (questions doc Q6). Held design (from the 2026-07-09 research): per-subject watermarks from meta.lastUpdated / Bulk $export?_since= when available, else per-bundle content hash; re-evaluate a (subject, measure) when data hash changed OR measure version changed OR compliance period rolled over; measure-logic/value-set/period changes are broad-rerun invalidation triggers; predicate pushdown builds only a CONSERVATIVE SUPERSET of Initial-Population candidates (SQL may SKIP work, never DECIDE outcomes — ADR-008 extension; the inInitialPopulation signal from PR #250 is the primitive). Likely needs a small (subject, measure, period, logicVersion, dataHash) eval-state table — SCHEMA IS OWNER-GATED: propose DDL in the design doc, do not create it without explicit approval.

What to do

Phase 1 (unblocked once Q6 answered): a design doc (docs/superpowers/specs/) with the watermark/hash scheme fitted to the actual API, the eval-state DDL proposal, invalidation matrix, and a parity plan (incremental run == full run on identical data).

Phase 2 (after owner approves design + DDL): implement behind the batch engine.

Acceptance criteria

• Design doc reviewed/approved by owner (incl. DDL sign-off) before any code
• Parity guarantee: for a fixed dataset, incremental == full-run outcomes (golden test in Phase 2)
• Skipped-subject accounting is auditable (run summary states evaluated vs skipped-unchanged)

How to verify

1. Confirm the Phase 1 design doc exists under docs/superpowers/specs/ and covers: the watermark/hash scheme fitted to the actual WebChart API, the proposed eval-state DDL ((subject, measure, period, logicVersion, dataHash)), the invalidation matrix, and the parity plan.
2. Confirm the design doc — including the DDL proposal — was reviewed and approved by the owner BEFORE any Phase 2 code was written.
3. In Phase 2, cd backend-ts && pnpm typecheck && pnpm test — must include a golden parity test proving incremental evaluation produces identical outcomes to a full run on a fixed, unchanged dataset.
4. Confirm the run summary accounts for skipped-unchanged subjects separately from evaluated ones (auditable).

Effort & dependencies

Effort: design 1–2 days; build sized after design. Dependencies: #254 (Issue 2, Q6); #256 (Issue 4, worker pool).

Hard rules (repo-wide, non-negotiable)

• CQL Outcome Status is the SOLE compliance authority (ADR-008). Nothing you build may set, override, or shortcut it.
• Every state change writes an audit_events row. No exceptions.
• NO new npm dependencies unless the issue explicitly approves one; schema/DDL changes are owner-gated (Taleef only — stop and ask).
• Verify before claiming done: cd backend-ts && pnpm typecheck && pnpm test (expect ~1057+ pass / 1 pg-skip / 0 fail) and, if frontend touched, cd frontend && npm run lint && npm run build.
• Update affected docs (ARCHITECTURE/DATA_MODEL/MEASURES/DECISIONS/DEPLOY + JOURNAL.md entry) in the same PR. Conventional commits. Feature branch + PR; owner merges.

Links

• docs/ROADMAP_2026-07-09.md (lands in the roadmap docs PR)
• docs/superpowers/specs/ (Phase 1 design doc)
• docs/MIE_INTEGRATION_QUESTIONS_2026-07-09.md (Q6)
• #250 (PR that introduced the inInitialPopulation signal)