koala73 / koala73/worldmonitor

🚨 forecast input funnel is too narrow + synthetic to validate the verification pipeline (Brier ≠ skill) (#4930/#5097)

Open
#5,233 1 comment 0 reactions 0 assignees View on GitHub
area: AI/intel bug
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## 🚨 The verification pipeline is measuring a biased, synthetic-heavy input funnel — Brier ≠ real forecasting skill

We built a working resolution + scorecard pipeline (#4930: resolver live, 5 scored, brier 0.28, judged-window fixed via #5196). **But the forecasts flowing into it are too narrow and too synthetic to validate whether the pipeline — or the model — is any good.** You cannot judge a forecasting system's skill on this sample.

### Evidence (prod, 27 unique forecasts over 21 hourly snapshots, 2026-07-12)
| Domain | count | notes |
| --- | --- | --- |
| market | 11 (41%) | **9 of 11 are `state_derived` synthetic backfill**, not detector predictions |
| supply_chain | 8 (30%) | detector — working feed (chokepoints) |
| cyber | 4 (15%) | detector — working feed |
| infrastructure | 3 (11%) | detector — working feed |
| conflict | 1 (4%) | intermittent |
| **political/unrest, energy, macro, health, maritime, displacement, naval, food** | **0** | no forecasts at all |

- **37% of the funnel is `state_derived` synthetic backfill** (10/27), concentrated in market.
- Only **5 domains** ever produce forecasts, and 3 of them are the ones that happen to have live resolution feeds. The geopolitical + macro + energy + health surface — the core of a "world monitor" — is **absent**.
- The scorecard scores this whole sample, so `brier`/`voidRate`/coverage are computed over synthetic-padded, 5-domain data.

### Root cause (two coupled, both generator-side)
1. **Narrow detector coverage (the *missing-signals* half of #5097).** The signal-integrity *bugs* are fixed (#5098 FRED, #5099 ACLED, #5101 observability — all closed). But #5097's missing-signals audit found ~20 feeds **collected but consumed by no detector** (EIA/JODI energy, FAO food, GDELT, displacement, disease, USNI naval, NOTAM…). No detector → no forecasts for those domains → the funnel is stuck at ~5 domains.
2. **Synthetic backfill to a fixed publish target.** `MIN/MAX_TARGET_PUBLISHED_FORECASTS = 10–14` (`seed-forecasts.mjs:116-117`); `selectDeferredForecastForPublishBackfill` (`:13862`) pads the detector shortfall with `state_derived` forecasts (`:1623`) — which is why market/synthetic dominates. These carry awkward auto-titles ("Maritime energy flow disruption from Black Sea maritime disruption state (market)") that also resolve poorly.

### Why this is the red alert
Every downstream conclusion — "brier 0.28", "vs-market skill", "≥80% resolvable" — is drawn from a sample that is **37% synthetic and missing the domains the product exists to forecast**. We could ship a perfect resolver and still learn nothing about real skill. This blocks the *meaning* of #4930's KPIs even though the plumbing works.

### Required actions
1. **Widen detector coverage** to the collected-but-unused signals (the #5097 missing-signals list — start with energy: EIA/JODI/inventories/SPR; then food-prices, GDELT, displacement). Each new detector adds a real domain to the funnel.
2. **Segregate synthetic from the skill metric.** The scorecard already slices `byGenerationOrigin` — **report `legacy_detector`-only Brier as the headline skill number**, exclude/flag `state_derived` backfill so it doesn't contaminate the measured skill. (Fast, high-value.)
3. **Curb / improve the backfill.** Don't pad Brier-scored output with low-quality synthetic forecasts to hit a count target; if backfilling for UX, mark them non-scored. Fix the auto-generated titles/claims so they're crisp and resolvable (see companion question-quality issue).
4. **Add a funnel-diversity guardrail** (log/alert): if a run publishes X% state_derived, that's a signal the funnel has collapsed.

### Done when
The scorecard's **detector-only** skill sample covers ≥6–8 domains with real geopolitical/macro/energy representation, `state_derived` is excluded from the headline Brier, and no run is majority-synthetic.

Related: #4930 (verification — this defines whether its KPIs mean anything), #5097 (generator integrity — the missing-signals half), #5091 (resolvable coverage), #5092 (Bet-5 track record — must not publish a synthetic-inflated Brier).

Contributor guide

Open the contributing guide

Research direction

Start in seed-forecasts.mjs, especially MIN/MAX_TARGET_PUBLISHED_FORECASTS (lines 116-117), selectDeferredForecastForPublishBackfill (line 13862), and state_derived creation (line 1623); inspect how the scorecard's byGenerationOrigin slice is produced. Trace the #5097 missing-signals list and existing detector inputs before choosing coverage additions. Done means detector-only scoring spans 6–8 domains, state_derived is excluded from headline Brier, and runs are not majority-synthetic.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
analytics, backend, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.