koala73 / koala73/worldmonitor
feat(forecast): empirical recalibration layer fitted on the published-origin resolved ledger
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
Part of #7066. Must land after #7071 establishes the post-market-blend input and after the resolution-loop fixes define a healthier forward population.
## Goal
Add an auditable empirical recalibration layer for published-origin forecasts, evaluate it out of sample, and activate it as the canonical displayed/scored probability only when forward evidence shows no material regression.
## Correct target population
The pooled n=313 scorecard contains 186 excluded shadow/synthetic entries and must not be used to fit the product map.
Published-origin baseline, n=127:
| Bucket | n | Predicted mean | Realized |
|---|---:|---:|---:|
| 0-10 | 33 | 0.061 | 0.000 |
| 10-20 | 25 | 0.145 | 0.000 |
| 20-30 | 12 | 0.262 | 0.000 |
| 30-40 | 19 | 0.358 | 0.368 |
| 40-50 | 16 | 0.449 | 0.250 |
| 50-60 | 17 | 0.532 | 0.059 |
| 60-70 | 5 | 0.610 | 0.200 |
| 70-100 | 0 | - | - |
Cyber supplies 96/127 published outcomes. Do not apply a cyber-dominated global map to sparse domains without validating cross-domain transfer.
## Probability lineage
Persist these distinct values:
1. `internalProbability`: detector/ensemble output before market anchoring.
2. `marketBlendedProbability`: result after #7071 and domain caps.
3. `calibratedProbability`: output of the empirical map.
4. `probability`: canonical displayed, projected, ledger-recorded, and scored value.
During shadow rollout, `probability` remains the market-blended value. After the activation gate passes, set `probability = calibratedProbability` and retain the earlier fields plus map version and `probabilitySource` for audit.
## Implementation
1. Add a pure monotone fitter in `scripts/_forecast-calibration.mjs`, using individual published-origin outcomes from the rolling window and injected timestamps.
2. Fit per domain only when its preregistered minimum sample is met. Sparse domains use identity. A hierarchical/global fallback is out of scope unless a time-separated validation proves non-regression for the receiving domain.
3. Version and persist the map, fit window, cohort filter, per-domain sample, source probability stage, fit timestamp, and code version.
4. Run the new map in shadow. Store both market-blended and calibrated values, but do not change the displayed/scored canonical value.
5. Extend the scorecard with ECE/reliability output and forward raw-vs-calibrated Brier. Use Wilson intervals for realized proportions and entry bootstrap for Brier/ECE uncertainty.
6. Activate only on a later, time-separated cohort. Never evaluate activation on the same outcomes used to fit the map.
## Activation gate
- At least 60 forward published-origin outcomes overall and at least 30 in any domain receiving a non-identity map.
- Paired bootstrap over entry-level Brier loss shows calibrated probability is non-inferior overall and in every activated domain.
- Coverage, VOID, and origin mix are reported beside skill so selection changes cannot masquerade as improvement.
- The map is monotone, versioned, and reversible to identity without losing raw probability lineage.
## Tests
- Deterministic golden fit on a frozen published-origin ledger.
- Monotonicity property test.
- Identity fallback for insufficient total/domain sample, with positive controls for both branches.
- Population filter excludes `bet_engine` and `state_derived`.
- Shadow mode preserves canonical `probability`; activated mode changes it and preserves lineage.
- Fit and evaluation cohorts cannot overlap.
- No live clock in fixtures.
## Acceptance
- [ ] #7071 is live and the fitter consumes the post-blend field.
- [ ] Published-origin-only fit metadata is inspectable.
- [ ] Sparse domains remain identity unless transfer is validated.
- [ ] Shadow raw/blended/calibrated scores are reported on a time-separated cohort.
- [ ] Activation gate passes before canonical `probability` changes.
- [ ] Every consumer of canonical probability, including projections and the resolution ledger, is tested after activation.
- [ ] Rollback to identity is documented and tested.
Contributor guide
Research direction
Read #7071 and the post-market-blend input path first, then inspect scripts/_forecast-calibration.mjs and the probability lineage consumers. Use the listed deterministic fit, monotonicity, fallback, population-filter, shadow/activation, and cohort-separation tests as the implementation checkpoints. Done means a versioned published-origin map is evaluated out of sample and activation, lineage, reporting, and identity rollback are tested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, typescript
- Domain
- analytics, backend, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100