koala73 / koala73/worldmonitor
feat(forecast): emission-time resolvability gate — dry-run the extractor before attaching a hard spec
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
Part of #7066 and direct follow-up to #5091.
## Goal
Prevent a hard resolution spec from being attached when the exact resolver path cannot extract its metric at emission time. Prove the gate in shadow before it redirects forecasts into the judged lane.
## Verified baseline
- 137/450 resolved entries are VOID; 120 are `no_establishable_metric`.
- Infrastructure VOID is 78/89, supply-chain VOID is 51/61, and political VOID is 3/3.
- `buildHardSpec` validates threshold and source-feed mapping but does not execute `extractMetricObservation`.
- Generator inputs and the resolver's `shapeResolutionFeed` view are not the same contract. The gate must use the exact resolver-shaped feed view.
- The judged lane is currently backlogged and must not be treated as a safe fallback until #7068/#7069 establish its SLO.
## Implementation
### 1. Share the exact extraction contract
Extract or export a pure feed-shaping seam used by both emission-time validation and the resolver. It must produce the same keyed input that `extractMetricObservation` receives at resolution time.
Do not call the extractor against a generic generator snapshot. Do not add network or clock reads. Preserve deterministic output for identical `(prediction, shapedInputs, generatedAt)`.
### 2. Shadow phase
For every candidate hard spec:
- execute `extractMetricObservation(candidateSpec, shapedFeed)`;
- record pass/fail, family, domain, metric key, and reason;
- keep the existing attached spec unchanged;
- emit per-family and per-domain counters in the run summary.
Run shadow mode until at least seven daily runs are recorded. Report the exact would-downgrade cohort and confirm that its destination lane can meet the judged-lane SLO.
### 3. Activation phase
Activation depends on #7068 and #7069:
- pending-judge is below 20;
- the queue has not grown across seven daily runs;
- first-attempt and resolved-within-SLA metrics are visible;
- no known archive-horizon stranding remains.
After the gate is enabled, a non-finite extraction downgrades to a judged spec with `specOrigin: hard_downgraded_unextractable`. Preserve the original hard family and failure reason for cohort follow-up.
If the judged-lane activation gate is not satisfied, remain in shadow. Do not improve hard-VOID numbers by moving failures into a broken lane.
### 4. Follow-through measurement
Track every downgraded `id@deadline` through maturity. Report its scored, VOID, pending, and resolution-latency outcome. Compare this with the historical hard-path outcome for the same family. A downgrade is successful only when it increases scored-within-SLA yield.
## Tests
- Shared feed-shaping parity: emission and resolver build the same extractor input.
- Positive control: a valid metric returns finite and stays hard.
- Negative control: an absent geography/record returns non-finite and is marked for downgrade.
- Shadow mode never changes the attached spec.
- Activation mode attaches the tagged judged fallback only when the judged-lane gate is satisfied.
- No network, clock, or mutation in the pure gate.
- Existing detector and resolver suites stay green.
## Acceptance
- [ ] The gate uses the exact resolver-shaped feed contract.
- [ ] Seven daily shadow runs and the would-downgrade cohort are posted.
- [ ] Activation is blocked until #7068/#7069 meet the stated judged-lane gate.
- [ ] Both gate directions and shadow/activation behavior are proven by tests.
- [ ] Downgraded forecasts are reconciled through maturity; success is scored-within-SLA, not disappearance from hard VOID.
- [ ] Go-forward results are separated from legacy damage that this gate cannot prevent.
## Non-goals
- No broad `seed-forecasts.mjs` refactor.
- No retirement of `JUDGED_DOMAINS` in this issue.
- No claim that emission-time extractability prevents temporal/source loss at deadline.
Contributor guide
Research direction
Start by locating buildHardSpec, extractMetricObservation, and shapeResolutionFeed, then compare the emission and resolver feed contracts. Run the shared feed-shaping, positive/negative control, shadow, activation, detector, and resolver tests mentioned in the issue. Done requires parity, deterministic shadow behavior, gated activation, and maturity outcome measurement across seven daily runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, data-engineering, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100