koala73 / koala73/worldmonitor

feat(scorecard): honesty markers — valid uncertainty, auditable corpus, replay-gated run diffs

Open
#7,072 0 comments 0 reactions 0 assignees View on GitHub
area:seeders enhancement P2
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

Part of #7066. #7074 and #6646 consume this corrected scorecard contract.

## Goal

Make the scorecard honest about cohort state, uncertainty, and evaluation limits. Reuse the existing append-only outcome receipts instead of building a second corpus.

## Problems to correct

1. `publicationCoverage = scored / entries` does not measure publication-to-ledger coverage. Its denominator already contains ledger entries and mixes immature, pending, VOID, and scored states.
2. A Wilson interval is valid for a binomial proportion, not a mean Brier loss.
3. A corpus containing only newly resolved entries has survivorship bias and drops pending/VOID failures.
4. Bare forecast IDs repeat across resolution windows; corpus identity is `id@deadline`.
5. `diff-forecast-runs.mjs` cannot claim candidate-code historical Brier skill unless both runs use the same frozen emission-time inputs and resolution windows. Matching a current stable ID to an old outcome is invalid.

## Scorecard schema

Replace or deprecate `publicationCoverage` with explicit fields:

- `publishedCount`: emission denominator for the cohort;
- `ledgerRegisteredCount` and `registrationCoverage`;
- `maturedCount`;
- `resolvedCount` and `resolvedWithinSlaCount`;
- `scoredCount` and `maturedScoredRate`;
- `voidCount` / `voidRate`, by reason;
- `pendingHardMatured` and `pendingJudgeMatured`;
- resolution-latency summary;
- population filters and rolling-window boundaries.

Keep aggregate, generation-origin, and domain slices. A count must accompany every accuracy/calibration value.

## Uncertainty contract

- Wilson 95% interval: realized rate in a calibration bucket, VOID proportion, hit rate, and other binomial proportions.
- Entry-level bootstrap 95% interval: mean Brier, log loss, ECE, and other mean losses.
- Paired entry-level bootstrap: WM-minus-market, raw-minus-calibrated, and candidate-minus-baseline Brier differences.
- For n=0, emit no estimate. For small n, emit the estimate, interval, and an explicit insufficient-sample flag.

## Evaluation corpus

The existing append-only R2 resolution receipt ledger is the source of truth. Add a versioned index/schema only where fields are missing.

Register every published forecast window before outcome filtering, keyed by `id@deadline`, with:

- emission time and code version;
- domain and generation origin;
- raw/internal, market-blended, calibrated, and canonical probability fields when present;
- resolution spec, deadline, and SLA;
- current state: immature, pending, resolved YES/NO, or VOID with reason;
- receipt key/hash without publishing private object paths.

Re-runs are idempotent by resolution-window key. Preserve state transitions; do not append a second logical forecast for the same window.

## Run-diff contract

Until #7073 proves exact replay:

- `diff-forecast-runs.mjs` continues to report structural differences;
- it may report the recorded corpus's historical Brier as context, clearly labeled as recorded predictions, not candidate-code skill;
- candidate Brier delta is `unavailable: emission_snapshot_required` unless both runs were produced from the same frozen emission snapshot and match the exact resolution-window corpus.

After replay exists, candidate comparison must use a frozen common-outcome subset and report excluded/missing windows.

## Tests

- Golden Wilson values for proportions, including n=0 and n=1.
- Deterministic seeded/bootstrap fixtures for Brier and paired Brier differences.
- State/denominator fixture covering immature, hard-pending, judge-pending, YES, NO, and each VOID class.
- Corpus idempotency by `id@deadline`, including the same stable ID at two deadlines.
- Run diff refuses a skill claim without a shared frozen snapshot and exact window identity.
- Existing hand-authored benchmark cases remain smoke tests only.

## Acceptance

- [ ] Misnamed `publicationCoverage` is removed/deprecated and the explicit funnel fields are emitted.
- [ ] Every estimate uses the correct interval family and includes n.
- [ ] The existing R2 receipt ledger is reused; no parallel outcome corpus is created.
- [ ] Every published/matured window remains visible, including pending and VOID entries.
- [ ] Corpus identity and idempotency use `id@deadline`.
- [ ] Run diffs state why candidate Brier is unavailable until exact replay exists.
- [ ] Consumers #7074 and #6646 are updated to the new names and semantics.

Contributor guide

Open the contributing guide

Research direction

Start with diff-forecast-runs.mjs and the existing append-only R2 resolution receipt ledger, then review the referenced consumers #7074 and #6646. Use the listed Wilson, bootstrap, state/denominator, corpus-idempotency, and run-diff fixtures to define completion; done means the explicit scorecard fields and uncertainty rules are emitted without a parallel corpus.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, typescript
Domain
analytics, data, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.