koala73 / koala73/worldmonitor
forecast: legacy 'unknown'-origin ledger entries can score into the skill-Brier headline (#5233 follow-up)
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
## Context
PR #5237 added a `skill` block to the forecast scorecard — a headline Brier computed over "real" generation origins, excluding synthetic (`state_derived`) and shadow (`bet_engine`). Greptile flagged (and we confirmed) a residual gap.
## The gap
`summarizeSkill` (`scripts/_forecast-scorecard.mjs`) maps a missing `generationOrigin` to `'unknown'`, which is **not** in the exclude set, so `unknown`-origin scored entries count toward `skill.count` / `skill.brier`.
The current pipeline tags entries correctly — `buildHistoryForecastEntry` (`scripts/seed-forecasts.mjs`) defaults `generationOrigin` to `'legacy_detector'`, so **new** ledger entries are never `unknown`. But the resolver ledger (`forecast:resolutions:v1`) is persistent with 180d retention, and **~52% of live entries are `'unknown'`** — legacy entries created before that default existed.
## Risk (low, transient)
- Today **0** `unknown` entries are scored (all are `pending` / `pending-judge`); they mostly VOID or age out.
- Residual risk: a legacy `unknown` entry resolves YES/NO before aging out → counts as "real skill", potentially masking a still-synthetic-heavy headline.
- These entries age out of the 180d window regardless.
## Options
1. **Monitor only** — the `byGenerationOrigin` slice already shows the `unknown` count; watch whether any `unknown` entries reach `scored`. (Cheapest; the population self-drains.)
2. **One-time backfill** — best-effort re-tag legacy `unknown` ledger entries from their forecast id / history where recoverable.
3. **Exclude `unknown` from the headline** — conservative, but risks understating real skill if any untagged entries are genuinely real.
Recommendation: (1) monitor; escalate to (2) only if `unknown` entries start scoring materially.
Refs #5233, #5237
Contributor guide
Research direction
Start with summarizeSkill in scripts/_forecast-scorecard.mjs and the generationOrigin default in buildHistoryForecastEntry in scripts/seed-forecasts.mjs; inspect the resolver ledger data described in the issue. Confirm whether any unknown-origin entries reach scored status and report the observed count and impact, following the monitor-only recommendation unless the issue's threshold for escalation is met.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- analytics
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100