koala73 / koala73/worldmonitor

epic(forecast): measurement substrate — close the resolution loop, recalibrate, fix the market blend

Open
#7,066 3 comments 0 reactions 0 assignees View on GitHub
area:seeders epic P1
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## Goal

Repair the forecasting measurement substrate before it is used to judge future engine changes. The current failures are real, but the old scorecard mixed measurement states and populations. This epic must improve the share of matured published forecasts that receive trustworthy outcomes without hiding failures by moving them between hard, judged, pending, and VOID states.

## Verified production baseline

Snapshot: 2026-08-22, from `forecast:scorecard:v1` and `forecast:resolutions:v1`.

| Signal | Baseline | Correct reading |
|---|---:|---|
| Ledger entries | 682 | Entries already registered in the resolution ledger |
| Matured entries | 551 | Deadline has passed |
| Resolved | 450 | 81.7% of matured entries |
| Scored YES/NO | 313 | 56.8% of matured entries |
| VOID | 137 | 30.4% of resolved entries |
| Matured hard-pending | 10 | Past deadline without a terminal outcome |
| Matured judge-pending | 91 | Past deadline without a terminal outcome |
| Old `publicationCoverage` | 45.9% (313/682) | This is scored-entry share, not publication-to-ledger coverage |
| Headline published-origin skill | Brier 0.125, n=127 | 186 shadow/synthetic scored entries excluded; cyber contributes 96/127 |
| Published-origin anchored cohort | WM 0.0957 vs market 0.0640, n=26 | The pooled n=41 includes 13 bet-engine and 2 state-derived entries |

VOID reasons: 120 `no_establishable_metric`, 9 `unsupported_window`, and 8 `judge_retry_exhausted`.

Published-origin calibration is the population relevant to the product:

| Bucket | n | Predicted mean | Realized |
|---|---:|---:|---:|
| 0-10 | 33 | 0.061 | 0.000 |
| 10-20 | 25 | 0.145 | 0.000 |
| 20-30 | 12 | 0.262 | 0.000 |
| 30-40 | 19 | 0.358 | 0.368 |
| 40-50 | 16 | 0.449 | 0.250 |
| 50-60 | 17 | 0.532 | 0.059 |
| 60-70 | 5 | 0.610 | 0.200 |
| 70-100 | 0 | - | - |

Do not use the pooled n=313 calibration table to fit or justify a published-product change.

## Shared measurement contract

All children must use the same contract:

1. Register every published forecast window before outcome filtering. Use the resolution-window identity `id@deadline`; a bare stable forecast ID is not unique across windows.
2. Freeze each forward evaluation cohort by emission time, code version, generation origin, domain, and deadline.
3. Reconcile every matured cohort entry as scored, VOID with a reason, hard-pending, or judge-pending. Never drop an entry from the denominator because it is difficult to resolve.
4. Publish separate metrics for:
- emitted-to-ledger registration coverage;
- resolved-within-SLA / matured;
- scored / matured;
- VOID / resolved, by reason;
- time to resolution;
- first-attempt judged sealing and attempts per resolved judged entry.
5. Report Brier and calibration by generation origin and domain. Use entry-level bootstrap intervals for mean Brier and paired bootstrap intervals for Brier differences. Wilson intervals are for binomial proportions only.
6. Preserve probability lineage: internal pre-market probability -> market-blended probability -> empirically calibrated probability -> canonical displayed/scored probability.
7. Use time-separated forward outcomes for activation decisions. Training-fit improvement is not acceptance.
8. Require no material regression in a supported domain. Sparse domains remain on the identity/no-change path until they have enough forward evidence.

## Sequence

### Safety and resolution loop

- [ ] #7068 - add attempt-level instrumentation and archive/judge safety controls.
- [ ] #7069 - freeze and safely drain the current backlog with canary, stop, rollback, and item-level reconciliation.
- [ ] #7068 - land the permanent first-attempt/archive-horizon repair and observe the judged-lane SLO.
- [ ] #7067 - run the emission-time extraction gate in shadow, then enable judged fallback only after the judged lane is healthy.

### Honest measurement

- [ ] #7072 - correct scorecard denominators and uncertainty, and index the existing audit ledger as the evaluation corpus.
- [ ] #7073 - in parallel, determine whether exact emission-time replay is possible.

### Probability changes

- [ ] #7071 - test and deploy the provisional 0.9 market weight on the matched published-origin cohort.
- [ ] #7070 - fit on the post-blend probability, run in shadow, then activate only after forward non-regression gates pass.

### Product surfaces

- [ ] #7074 - show the corrected scorecard in `ForecastPanel`.
- [ ] #7075 - add preregistered per-horizon outcome contracts and observers before scoring projections.

## Epic acceptance

- [ ] The old `publicationCoverage` field is removed or renamed; all four coverage/yield measures above are available.
- [ ] A frozen post-fix cohort is reconciled entry by entry with no silent omissions.
- [ ] All matured published entries reach a terminal outcome within the documented SLA, or remain explicitly listed as SLA breaches.
- [ ] Total VOID and unresolved-after-maturity improve without an offsetting increase in another failure state.
- [ ] Raw, market-blended, and calibrated probabilities are scored separately on forward outcomes.
- [ ] Market-relative results use only the matched published-origin cohort and paired uncertainty.
- [ ] No child claims success from in-sample fit, renamed failures, queue-depth reduction alone, or a changed cohort mix.
- [ ] The panel/public surfaces state sample size, population exclusions, freshness, and whether projections are scored.

## Scope boundaries

- Do not create a second outcome corpus; extend/index the existing append-only R2 receipt ledger.
- Do not perform opportunistic extraction from `seed-forecasts.mjs`. Extract only where needed to share a pure contract or test the requested behavior.
- Do not promote `bet_engine` or enable a learned calibration map until its explicit empirical gate passes.
- Merge and deployment remain separate decisions.

## Related

- #5091 - resolvability KPI; superseded measurement details live in #7067/#7068.
- #6646 - public scorecard page; depends on corrected scorecard semantics and an explicit anonymous route contract.
- #5007 - original resolver/scorecard delivery; remaining acceptance is owned by this epic.
- #5525 - implementation complete, but its empirical Gate 2 did not pass; promotion remains off.

Contributor guide

Open the contributing guide

Research direction

Start with the existing forecast:scorecard:v1 and forecast:resolutions:v1 outputs and the append-only R2 receipt ledger; review the linked child issues (#7067–#7075) to identify the relevant entry points. Do not use seed-forecasts.mjs for opportunistic extraction. Done means the frozen cohort is reconciled without omissions, coverage and resolution metrics are separated, probability lineages are preserved, and the epic acceptance checks pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
analytics, backend, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.