epic(forecast): verified probabilistic forecasting — resolution engine, public Brier track record, learned calibration, LLM ensemble (MiroFish++ roadmap)

# epic(forecast): verified probabilistic forecasting — resolution engine, public track record, learned calibration, LLM ensemble

## Why this is a bet

The forecast system (MiroFish-inspired) is architecturally ahead of most "AI prediction" products — 10 rule-based detectors fused over ~40 live Redis inputs, Polymarket/Kalshi anchoring, a theater-limited multi-agent simulation that rescores forecasts, and even a self-refining prompt loop with regression rollback. But **nothing ever checks whether a forecast came true**. Repo-wide: no resolution, no Brier score, no accuracy ledger. `forecast:predictions:history:v1` stores 45 days of snapshots that are never scored against reality. The existing "calibration" field is market-price anchoring, not verification; `evaluate-forecast-run.mjs` / `evaluate-forecast-benchmark.mjs` are structural/generation QA only.

That missing loop is simultaneously our biggest credibility risk ("AI slop probabilities") and the biggest available selling point: **no consumer/prosumer intelligence product publishes a scored, auditable forecasting track record.** WorldMonitor can, because — unlike MiroFish or a chat wrapper — we already operate the ground-truth sensors (ACLED/UCDP, PortWatch chokepoints, market/commodity quotes, EIA/JODI energy, outages, GPS jamming, sanctions...). The same feeds that generate a forecast can resolve it automatically.

Context: state of the art moved fast — LLM ensembles now approach superforecaster Brier parity on ForecastBench (projected parity late 2026; AIA Forecaster already matches on market-derived questions). The moat is shifting from "can you forecast" to "can you prove it and wire it into live data." We have the second part.

## ✅ Status & forward plan — updated 2026-07-09 (supersedes earlier status + comment thread)

The verification loop is **live in production.** Bets 1–2 (deterministic half) are shipped and deployed; the scorecard is accumulating. What remains is (a) making every day's output fully *resolvable* and (b) waiting — with instrumentation — for enough resolved forecasts to unlock the learning/product bets.

| Bet / piece | State |
| --- | --- |
| **Bet 1** resolvability contract | ✅ shipped & live (#4976→#5004; amendment #5010→#5012) |
| **Bet 2** resolver + scorecard (core) | ✅ shipped (#5024), **deployed** as the `seed-forecast-resolutions` Railway daily cron (#5062) |
| ↳ count-family stale-feed safety | ✅ #5063 (lagged/capped feeds → `pending`/`VOID`, never false-NO) |
| ↳ coverage: UCDP→ACLED repoint + political/cyber promoted to hard + publish boost | ✅ #5072 (closes #5066) |
| ↳ ledger retention/pruning | ✅ #5073 (closes #5067) |
| ↳ **judged resolution** (dual-model LLM + citations + VOID) | ⏳ **not built** — tracked in #5007, **the #1 do-now** |
| **Bets 3–6** | not started (data-gated or parallel — see lanes) |

**Live as of 2026-07-09:** scorecard `entries: 55, scored: 0, pending: 20, pending-judge: 34, void: 1`; canonical mix **50% hard / 50% judged** (≥80%-hard KPI not yet met — likely supply-limited, needs more judged→hard). **`scored: 0`** because forecasts haven't matured yet, and **62% of the ledger is `pending-judge`, which cannot score until judged-resolution ships.**

### The reframe: "waiting for data" gates only three things

Waiting only blocks **Bet 3**, **Bet 4-promote**, and **Bet 5-launch**. Everything else is do-now — and it determines whether the accumulating data is worth anything. Shipping judged-resolution *now* vs *in 30 days* is the difference between accumulating at ~90% vs ~50% coverage for a month, and that month is unrecoverable. The wait is a window to make the pipeline good before it fills.

### 🟢 Lane A — do NOW (makes the accumulating data usable) · tracked: **#5091 #5092 #5093 #5094**
1. **Judged resolution (#5007) — #1 priority.** ✅ **delivered via #5087, deployed 07-09.** Without it 62% of every day's forecasts never score; waiting cannot fix it, only the judge can.
2. **Lock the ≥80%-hard KPI → #5091.** Live **57%** (was 43%); #5072 helped but the KPI is still unmet — supply-limited, so promote more detectors judged→hard. **Most time-sensitive:** forecasts published under-covered are *permanently* under-scored while the wait runs.
3. **Bet 4 ensemble — build the *shadow* now → #5093.** Shadow-publish an LLM-ensemble probability beside the detector probability so it's scored alongside from day one (promotion is Lane C).
4. **Bet 5 — build the page *shell* now → #5092.** `/forecasts/track-record` reading the live `get-forecast-scorecard`; honestly shows "N resolved, accumulating." Data-independent to build; only the public launch is gated.
5. **Bet 6 sim verification / persistent actor memory → #5094.** No data dependency — fully parallel.
6. **Observability.** ✅ **done 07-09** — scorecard-watch (local script + daily cloud routine) fires the two first-fire milestones (first hard scored / first judged sealed) plus resolver-stall & VOID-spike alerts.

### 🟡 Lane B — the wait (watch, don't guess dates)
Resolver runs daily 06:00 UTC. Short-horizon at-deadline hard families (`price`/`yesPrice`/`hexCount`) should throw the **first real scored YES/NO within ~1–2 weeks**; ACLED counts lag by their settlement window. **The scorecard `resolved`/`scored` count is the progress bar — trigger on it, not the calendar.**

### 🔴 Lane C — wait-gated (unlock on data triggers, not dates)
| Bet | Unlock trigger |
| --- | --- |
| **Bet 3** learned calibration | ≥~30 days **and** ≥N resolved *per domain* |
| **Bet 4** promote LLM blend over detector prior | shadow blend beats detector on rolling Brier (needs Lane-A #3 live) |
| **Bet 5** public launch | ≥100 resolved **and** VOID rate <15% |

### This week
Ship **judged resolution (#5007)**, confirm coverage climbs toward ≥80%, stand up the scorecard-watch. Then: build Lane A in parallel, watch Lane B fill, trigger Lane C on data.

_Also open, off the critical path: #4933 (pre-#4930 hardening residual), #4963 (Brier baseline not LLM-free for market/supply_chain), #5007 remaining deliverables._

## Current state (mapped 2026-07-06)

- **Generator**: `scripts/seed-forecasts.mjs` (17.8k lines), hourly Railway seeder. Detectors `detectConflictScenarios` (:982) … `detectFromPredictionMarkets` (:2163) → hand-tuned probabilities with per-domain caps → market blend `0.4*market + 0.6*internal` → hard-coded horizon projection curves → LLM narrative enrichment (Groq `llama-3.1-8b-instant` → OpenRouter `gemini-2.5-flash`; LLM never sets probabilities) → publish 10–14 to `forecast:predictions:v2`.
- **MiroFish sim**: "Phase 2 — Theater-Limited Simulation Runner" (:16536) — per-theater 2-round LLM sim (3 fixed archetypes: escalation/containment/market_cascade, temp 0) → `applySimulationMerge` (:11613) adjusts path acceptance (+0.08×conf bucket-channel match, +0.04 actor overlap, −0.12 invalidator, −0.15 stabilizer), applied with one-cycle delay via `forecast:sim-decorations:v1`.
- **Learning that exists**: impact-expansion prompt self-refinement (`forecast:prompt:impact-expansion:learned`, locked scorer + rollback, :15694) — refines *prose*, not *probability truth*.
- **Separate deterministic scenario engine**: `run-scenario` supply-chain worker (6 templates, exposure×disruption×costShock math) — currently disconnected from forecasts.
- **Verification**: none (the gap this epic closes).

## The roadmap — 6 bets, strict spine 1→2→3

### Bet 1 — Resolvability contract (foundation, ~1 wk)
Every published forecast carries a machine-checkable resolution spec, emitted by the same detector that scored it:

```
resolution {
kind: hard | judged
metric_key // e.g. redis feed + JMESPath: chokepoints v4 hormuz.transitDelta
operator/threshold, window
deadline // derived from time_horizon at publish
source_feed // provenance: which WM feed resolves it
}
```

- Hard-resolvable classes (target ≥80% of published): conflict event counts (ACLED/UCDP), market/commodity levels (quotes feeds), chokepoint transit deltas (PortWatch), outage counts, GPS-jamming intensity, sanctions-list changes, theater posture transitions.
- `judged` fallback for the rest: resolution question text + retrieval scope, resolved by LLM judge in Bet 2 (dual-model agreement + citations, else VOID).
- Proto: add `Resolution` to `proto/worldmonitor/forecast/v1/forecast.proto`; extend `scoreForecastReadiness` so an unresolvable forecast is publish-penalized. Side effect: forces sharper question titles (today's "Escalation risk in X" vagueness is exactly what makes verification impossible).
- **New module, not more monolith** — seed-forecasts.mjs is 17.8k lines; resolution spec builders live in a new `scripts/_forecast-resolution.mjs` imported by the seeder.

### Bet 2 — Resolution engine + scorecard (the verification loop, 1–2 wk)
New seeder `seed-forecast-resolutions.mjs` (daily cron; one `runSeed()` per process):
- Sweep `forecast:predictions:history:v1` for matured forecasts → read `metric_key` from the same Redis feeds → resolve YES/NO/VOID; `judged` ones via two independent models over WM news archive with mandatory citations, disagreement → VOID.
- Write per-forecast outcomes to `forecast:resolutions:v1` + append-only R2 ledger (audit trail with the evidence snapshot at resolution time).
- Aggregate `forecast:scorecard:v1`: rolling Brier + log score, per-domain/per-detector calibration curves, **vs-market skill** (on market-anchored forecasts: did internal beat the Polymarket close?), sample counts.
- Anti-gaming invariants: every published forecast is scored (no cherry-picking); VOID rate reported; scorer code path locked + tested like the existing benchmark suite.
- Start this clock ASAP — public credibility (Bet 5) needs ≥100 resolved forecasts (~30–60 days at 10–14/hr-run with dedup across runs).

### Bet 3 — Learned calibration replaces hand-tuned constants (after ~30d of resolutions, ~1 wk)
Repurpose the existing locked-scorer/rollback discipline from prompt-learning to probability-learning:
- Per-detector reliability: isotonic/Platt post-calibration with low-N shrinkage → replaces static domain caps (cyber 0.72, conflict 0.90...) as data accumulates.
- Market-blend weight learned per domain (global 0.4/0.6 today).
- Sim-merge weights (+0.08/−0.12/−0.15) re-fit against which sim-backed paths actually materialized.
- Horizon projection curves (today hard-coded per domain) replaced by empirical decay measured from resolutions.
- Guardrail: frozen-baseline shadow scoring; auto-rollback if rolling Brier regresses (same pattern as `PROMPT_LEARNED_KEY` rollback).

### Bet 4 — LLM forecaster ensemble on the top stories (after Bet 2, 1–2 wk, flag-gated)
Today the LLM never sets a probability — that leaves the strongest known accuracy lever unused (retrieval-grounded LLM ensembles approach superforecaster parity on ForecastBench). For the top K≈6 forecasts per run:
- 3 diverse reasoning passes over the already-built evidence table + news context + theater-sim paths: outside-view/base-rate, inside-view/signal-weighing, adversarial refuter.
- Trimmed-mean aggregate → three-way blend: detector prior + LLM ensemble + market anchor (weights learned in Bet 3).
- Stronger model for this stage only (~18 calls/run, prompt-hash cached like every other stage; respects the per-run LLM budget).
- Prove it before promoting: shadow-publish LLM-blended probability alongside detector probability; Bet 2 scores both; promote when the blend wins on rolling Brier.

### Bet 5 — Track record as product (parallel after Bet 2, 1–2 wk build, publish when N≥100)
The selling point:
- Public `/forecasts/track-record`: rolling Brier + calibration plot, per-domain breakdown, vs-market skill, and a **resolved-forecast ledger with receipts** (what we said, when, at what probability → what happened → resolving source). "The only intelligence dashboard that publishes its Brier score."
- Forecast cards: detector reliability badge + family resolution history; "Resolved ✓/✗" chips.
- API/MCP: resolution fields on `get-forecasts`, new `get_forecast_track_record` tool — agents can audit accuracy before trusting (agent-readiness/orank alignment).
- Brief/digest hook: "forecast movements & resolutions" section (reuses `annotateForecastChanges`).
- Gating: panel stays free-trial; track-record summary public (it IS the marketing); deep sim, conditional scenarios, per-detector analytics → Pro.

### Bet 6 — Simulation that earns its keep (parallel after Bet 1, incremental)
- **Verifiable sim**: theater invalidators/stabilizers/timingMarkers are already structured — bind them to feed metrics so branch predictions get resolved and scored like forecasts (sim gets its own track record; kills "narrative theater" criticism).
- **Persistent actor memory**: actor postures/objectives evolve across runs (continuity records exist; today each run mostly resets) — the genuinely valuable MiroFish idea.
- **Bridge the two engines**: a forecast with `routeFacilityKey` auto-runs the deterministic supply-chain worker → "if this lands: top exposed countries/sectors" card with real trade math. Demo-able differentiator; zero new data needed.
- **Pro custom scenarios**: parameterized user-triggered theater sims via the existing `trigger-simulation` queue (bounded templates, not free-form).

## KPIs
- ≥80% of published forecasts carry hard resolution specs (Bet 1)
- 100% of matured forecasts resolved ≤24h after deadline; VOID rate <15% (Bet 2)
- Rolling 90d Brier <0.20 at publish, target <0.15; calibration slope→1 (Bets 3–4)
- Vs-market skill ≥ parity on anchored forecasts (beating Polymarket is the headline if achieved)
- ≥100 resolved forecasts before public track-record launch (Bet 5)

## Risks / gotchas
- **Self-grading bias**: prefer hard metric resolution; judged resolutions need dual-model agreement + citations + audit ledger; VOID over guessing.
- **Question quality**: resolvability forces title sharpening — expect detector title rewrites (good, but touches published copy).
- **Monolith growth**: all new stages as separate modules/seeders; seed-forecasts.mjs must not grow another 5k lines. One `runSeed()` per process (seeder gotcha).
- **Cost**: resolver is deterministic reads; LLM ensemble bounded to top-K with prompt-hash caching per existing cache-identity discipline.
- **Public API exposure**: track-record endpoints are anon-read → cache headers + client-caller sweep per the premium-gating 4xx-flood pattern.

## References (state of the art)
- ForecastBench (dynamic AI forecasting benchmark; superforecaster Brier 0.081 vs best-LLM ~0.10, parity projected late 2026)
- AIA Forecaster: matches superforecasters on market-derived questions (FB-Market 0.0753 vs human 0.0740)
- Eternis: calibration-aware reward design (asymmetric penalty for confident-wrong) as the training/eval principle — we apply the same asymmetry in scorecard weighting
- MiroFish (github.com/666ghj/MiroFish): GraphRAG + agent personas + parallel simulation + report agent — our theater sim is the pragmatic subset; Bet 6 adopts its persistent-memory idea without the thousand-agent cost

Treat this as a roadmap epic and choose an unfinished linked lane rather than editing the completed work. Read scripts/seed-forecasts.mjs, proto/worldmonitor/forecast/v1/forecast.proto, and the relevant issue such as #5091–#5094; run the existing forecast evaluation or scorecard checks first. Done means the selected lane is implemented, tested, and its stated data or publication trigger is met.

Written by the indexing model from the issue text.