koala73 / koala73/worldmonitor
feat(forecast): score the 24h/7d/30d projections — per-horizon observations and skill
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
Part of #7066. Lowest-priority child. Depends on #7067's shared extraction path and #7072's correct uncertainty helpers.
## Goal
Replace unfalsifiable 24h/7d/30d projection decoration with preregistered, horizon-specific probability and outcome contracts, then accumulate forward observations and score only eligible projections.
## Current limitation
`PROJECTION_CURVES` applies fixed editorial multipliers to one forecast probability. The resolution ledger has one spec with one event, threshold/window, and deadline. Three observations do not automatically create three valid binary outcomes, and old projections were not emitted with horizon-specific event contracts.
Do not retrospectively score historical projections that lack these contracts.
## Horizon contract
At emission, persist one `ProjectionResolutionSpec` per horizon, keyed by `id@deadline@horizon`:
- `horizon`: `h24` | `d7` | `d30`;
- emitted projected probability;
- `semantics`: `point_in_time` | `cumulative_by_deadline`;
- metric key/source feed;
- operator and threshold;
- baseline when required;
- observation window and horizon deadline;
- source-specific sample tolerance;
- emission time, code version, and parent resolution-window key.
The generator must explicitly select semantics from the claim/spec type. Cumulative events must satisfy probability-law constraints across horizons. Point-in-time state claims may move either direction but require separate observation windows.
If a claim cannot produce a complete horizon contract, mark that projection `unscored` and state the reason. Do not synthesize an outcome from the parent forecast's terminal result.
## Observation lifecycle
1. Register all three horizon tasks at emission.
2. Keep each task active until its own deadline even if the parent forecast resolves earlier.
3. At each deadline, use the shared #7067 resolver-shaped extraction path and the stored horizon spec.
4. Choose the nearest eligible sample inside the stored tolerance. No sample means `unobserved`, not NO.
5. Preserve late-arriving source behavior and a bounded finalization SLA.
6. Key idempotently by `id@deadline@horizon`.
V1 includes hard/extractable projections only. Judged horizon outcomes are out of scope.
## Scorecard
For each horizon and supported domain report:
- registered, matured, observed, unobserved, YES/NO, and VOID counts;
- Brier with entry-bootstrap interval;
- hit/realized rate with Wilson interval;
- point-in-time and cumulative contracts in separate slices;
- population and projection-curve version.
Do not fit new curve multipliers in this issue. Measurement must accrue first. Existing editorial curves remain visible as unvalidated priors until a later, separately gated learning issue.
## Tests
- Complete point-in-time and cumulative contracts produce expected outcomes.
- Cumulative probability-law violation is rejected at emission.
- Three tasks survive parent resolution and finalize independently.
- Sample inside tolerance resolves; nearest sample outside tolerance is unobserved.
- Same stable forecast at two deadlines and three horizons creates six distinct keys.
- Re-run is idempotent.
- Historical projections without contracts remain unscored.
## Acceptance
- [ ] Every newly scoreable projection carries a complete preregistered horizon contract.
- [ ] Horizon tasks continue after parent resolution and finalize independently.
- [ ] Unobserved is distinct from NO/VOID and remains in the coverage denominator.
- [ ] Scorecard reports horizon skill, uncertainty, contract semantics, and n.
- [ ] No historical backfill is scored without an emission-time contract.
- [ ] `PROJECTION_CURVES` are not fitted or promoted by this issue.
- [ ] `ForecastPanel` can distinguish scored from unscored projections.
Contributor guide
Research direction
Trace the ForecastPanel, PROJECTION_CURVES, and the shared resolver-shaped extraction path from #7067, then review the uncertainty helpers from #7072. Use the listed horizon-contract and lifecycle tests as the starting validation points. Done means eligible projections score independently per horizon, unobserved remains distinct from NO/VOID, historical projections without contracts stay unscored, and the scorecard reports the required slices and intervals.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- analytics, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100