koala73 / koala73/worldmonitor
feat(forecast): Bet 2 — resolution engine + Brier scorecard: score every forecast against ground truth (#4930)
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
# Bet 2 - resolution engine and Brier scorecard
## Status
The core implementation shipped: resolution specs, hard and dual-judge resolution, Redis working ledger, append-only R2 receipts, scorecard aggregation, and the scorecard RPC exist. This issue remains open for the original empirical acceptance contract, now owned by #7066 and its children.
## Live state, 2026-08-22
- 682 ledger entries; 551 matured; 450 resolved; 313 scored.
- 137 VOID (30.4% of resolved).
- 144 pending judge, including 91 past deadline.
- 120 VOID are `no_establishable_metric`.
- Published-origin headline Brier is 0.125 at n=127.
The engine exists, but the original "score every forecast" and VOID targets have not passed.
## Delivered
- [x] `ResolutionSpec` attached at publication.
- [x] Daily resolver for hard and judged specs.
- [x] Dual-model agreement and mandatory citations for judged outcomes.
- [x] `forecast:resolutions:v1` working ledger.
- [x] Append-only R2 resolution receipts.
- [x] `forecast:scorecard:v1` with Brier/log loss, domains/origins, calibration, market comparison, counts, and VOID.
- [x] Scorecard RPC and deployment wiring.
## Corrected invariants
1. Register every published forecast window before outcome filtering.
2. Use `id@deadline` as the outcome identity. A stable forecast ID alone repeats across evolving resolution windows.
3. Preserve every matured entry in the evaluation population, including pending and VOID states.
4. Score the probability actually shown for that window and preserve internal/market/calibrated lineage.
5. Keep R2 receipts as the single auditable outcome source; do not create a parallel corpus.
6. Report coverage/yield and skill together so survivorship changes cannot improve the score silently.
7. Never re-run detector code over history and call it comparable unless #7073 proves exact emission-time replay.
## Remaining work
- [ ] #7067: exact emission-time extractor gate.
- [ ] #7068/#7069: permanent judged-lane repair and safe backlog recovery.
- [ ] #7072: correct scorecard denominators, interval methods, corpus index, and replay-gated run diff.
- [ ] #7071/#7070: separately score internal, market-blended, and calibrated probabilities on forward outcomes.
- [ ] #6646/#7074: publish honest aggregate results after the contract is corrected.
## Empirical acceptance
- [ ] 100% emitted-to-ledger registration for the frozen cohort.
- [ ] Every matured published window reaches a terminal outcome within the documented SLA or is explicitly reported as an SLA breach.
- [ ] VOID rate is below 15% on a forward post-fix cohort, with reasons and domain slices.
- [ ] No silent scoring gaps; pending/VOID entries remain in denominators.
- [ ] Rolling Brier and calibration use published-origin outcomes and state n/uncertainty.
- [ ] Market-relative skill uses a matched published-origin cohort and paired uncertainty.
- [ ] Append-only receipts reconcile exactly with scorecard totals.
## Closure
Close this issue when the #7066 forward cohort satisfies the empirical acceptance above. Code delivery alone is already complete; production acceptance is not.
Part of #4930. Superseding execution epic: #7066.
Contributor guide
Research direction
Start with the superseding execution epic #7066 and its child issues #7067–#7074; the delivered resolution engine and scorecard are described here, but the remaining work is empirical acceptance. Review the acceptance checklist and close this issue only when the forward cohort meets its registration, terminal-outcome, VOID, denominator, scoring, and receipt-reconciliation requirements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- redis, typescript
- Domain
- analytics, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100