koala73 / koala73/worldmonitor

forecast: legacy 'unknown'-origin ledger entries can score into the skill-Brier headline (#5233 follow-up)

Open
#5,240 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## Context

PR #5237 added a `skill` block to the forecast scorecard — a headline Brier computed over "real" generation origins, excluding synthetic (`state_derived`) and shadow (`bet_engine`). Greptile flagged (and we confirmed) a residual gap.

## The gap

`summarizeSkill` (`scripts/_forecast-scorecard.mjs`) maps a missing `generationOrigin` to `'unknown'`, which is **not** in the exclude set, so `unknown`-origin scored entries count toward `skill.count` / `skill.brier`.

The current pipeline tags entries correctly — `buildHistoryForecastEntry` (`scripts/seed-forecasts.mjs`) defaults `generationOrigin` to `'legacy_detector'`, so **new** ledger entries are never `unknown`. But the resolver ledger (`forecast:resolutions:v1`) is persistent with 180d retention, and **~52% of live entries are `'unknown'`** — legacy entries created before that default existed.

## Risk (low, transient)

- Today **0** `unknown` entries are scored (all are `pending` / `pending-judge`); they mostly VOID or age out.
- Residual risk: a legacy `unknown` entry resolves YES/NO before aging out → counts as "real skill", potentially masking a still-synthetic-heavy headline.
- These entries age out of the 180d window regardless.

## Options

1. **Monitor only** — the `byGenerationOrigin` slice already shows the `unknown` count; watch whether any `unknown` entries reach `scored`. (Cheapest; the population self-drains.)
2. **One-time backfill** — best-effort re-tag legacy `unknown` ledger entries from their forecast id / history where recoverable.
3. **Exclude `unknown` from the headline** — conservative, but risks understating real skill if any untagged entries are genuinely real.

Recommendation: (1) monitor; escalate to (2) only if `unknown` entries start scoring materially.

Refs #5233, #5237

Contributor guide

Open the contributing guide

Research direction

Start with summarizeSkill in scripts/_forecast-scorecard.mjs and the generationOrigin default in buildHistoryForecastEntry in scripts/seed-forecasts.mjs; inspect the resolver ledger data described in the issue. Confirm whether any unknown-origin entries reach scored status and report the observed count and impact, following the monitor-only recommendation unless the issue's threshold for escalation is met.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
analytics
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.