koala73 / koala73/worldmonitor
feat(scorecard): honesty markers — valid uncertainty, auditable corpus, replay-gated run diffs
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
Part of #7066. #7074 and #6646 consume this corrected scorecard contract.
## Goal
Make the scorecard honest about cohort state, uncertainty, and evaluation limits. Reuse the existing append-only outcome receipts instead of building a second corpus.
## Problems to correct
1. `publicationCoverage = scored / entries` does not measure publication-to-ledger coverage. Its denominator already contains ledger entries and mixes immature, pending, VOID, and scored states.
2. A Wilson interval is valid for a binomial proportion, not a mean Brier loss.
3. A corpus containing only newly resolved entries has survivorship bias and drops pending/VOID failures.
4. Bare forecast IDs repeat across resolution windows; corpus identity is `id@deadline`.
5. `diff-forecast-runs.mjs` cannot claim candidate-code historical Brier skill unless both runs use the same frozen emission-time inputs and resolution windows. Matching a current stable ID to an old outcome is invalid.
## Scorecard schema
Replace or deprecate `publicationCoverage` with explicit fields:
- `publishedCount`: emission denominator for the cohort;
- `ledgerRegisteredCount` and `registrationCoverage`;
- `maturedCount`;
- `resolvedCount` and `resolvedWithinSlaCount`;
- `scoredCount` and `maturedScoredRate`;
- `voidCount` / `voidRate`, by reason;
- `pendingHardMatured` and `pendingJudgeMatured`;
- resolution-latency summary;
- population filters and rolling-window boundaries.
Keep aggregate, generation-origin, and domain slices. A count must accompany every accuracy/calibration value.
## Uncertainty contract
- Wilson 95% interval: realized rate in a calibration bucket, VOID proportion, hit rate, and other binomial proportions.
- Entry-level bootstrap 95% interval: mean Brier, log loss, ECE, and other mean losses.
- Paired entry-level bootstrap: WM-minus-market, raw-minus-calibrated, and candidate-minus-baseline Brier differences.
- For n=0, emit no estimate. For small n, emit the estimate, interval, and an explicit insufficient-sample flag.
## Evaluation corpus
The existing append-only R2 resolution receipt ledger is the source of truth. Add a versioned index/schema only where fields are missing.
Register every published forecast window before outcome filtering, keyed by `id@deadline`, with:
- emission time and code version;
- domain and generation origin;
- raw/internal, market-blended, calibrated, and canonical probability fields when present;
- resolution spec, deadline, and SLA;
- current state: immature, pending, resolved YES/NO, or VOID with reason;
- receipt key/hash without publishing private object paths.
Re-runs are idempotent by resolution-window key. Preserve state transitions; do not append a second logical forecast for the same window.
## Run-diff contract
Until #7073 proves exact replay:
- `diff-forecast-runs.mjs` continues to report structural differences;
- it may report the recorded corpus's historical Brier as context, clearly labeled as recorded predictions, not candidate-code skill;
- candidate Brier delta is `unavailable: emission_snapshot_required` unless both runs were produced from the same frozen emission snapshot and match the exact resolution-window corpus.
After replay exists, candidate comparison must use a frozen common-outcome subset and report excluded/missing windows.
## Tests
- Golden Wilson values for proportions, including n=0 and n=1.
- Deterministic seeded/bootstrap fixtures for Brier and paired Brier differences.
- State/denominator fixture covering immature, hard-pending, judge-pending, YES, NO, and each VOID class.
- Corpus idempotency by `id@deadline`, including the same stable ID at two deadlines.
- Run diff refuses a skill claim without a shared frozen snapshot and exact window identity.
- Existing hand-authored benchmark cases remain smoke tests only.
## Acceptance
- [ ] Misnamed `publicationCoverage` is removed/deprecated and the explicit funnel fields are emitted.
- [ ] Every estimate uses the correct interval family and includes n.
- [ ] The existing R2 receipt ledger is reused; no parallel outcome corpus is created.
- [ ] Every published/matured window remains visible, including pending and VOID entries.
- [ ] Corpus identity and idempotency use `id@deadline`.
- [ ] Run diffs state why candidate Brier is unavailable until exact replay exists.
- [ ] Consumers #7074 and #6646 are updated to the new names and semantics.
Contributor guide
Research direction
Start with diff-forecast-runs.mjs and the existing append-only R2 resolution receipt ledger, then review the referenced consumers #7074 and #6646. Use the listed Wilson, bootstrap, state/denominator, corpus-idempotency, and run-diff fixtures to define completion; done means the explicit scorecard fields and uncertainty rules are emitted without a parallel corpus.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, typescript
- Domain
- analytics, data, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100