koala73 / koala73/worldmonitor

feat(forecast): empirical recalibration layer fitted on the published-origin resolved ledger

Open
#7,070 0 comments 0 reactions 0 assignees View on GitHub
area: AI/intel feature P1
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

Part of #7066. Must land after #7071 establishes the post-market-blend input and after the resolution-loop fixes define a healthier forward population.

## Goal

Add an auditable empirical recalibration layer for published-origin forecasts, evaluate it out of sample, and activate it as the canonical displayed/scored probability only when forward evidence shows no material regression.

## Correct target population

The pooled n=313 scorecard contains 186 excluded shadow/synthetic entries and must not be used to fit the product map.

Published-origin baseline, n=127:

| Bucket | n | Predicted mean | Realized |
|---|---:|---:|---:|
| 0-10 | 33 | 0.061 | 0.000 |
| 10-20 | 25 | 0.145 | 0.000 |
| 20-30 | 12 | 0.262 | 0.000 |
| 30-40 | 19 | 0.358 | 0.368 |
| 40-50 | 16 | 0.449 | 0.250 |
| 50-60 | 17 | 0.532 | 0.059 |
| 60-70 | 5 | 0.610 | 0.200 |
| 70-100 | 0 | - | - |

Cyber supplies 96/127 published outcomes. Do not apply a cyber-dominated global map to sparse domains without validating cross-domain transfer.

## Probability lineage

Persist these distinct values:

1. `internalProbability`: detector/ensemble output before market anchoring.
2. `marketBlendedProbability`: result after #7071 and domain caps.
3. `calibratedProbability`: output of the empirical map.
4. `probability`: canonical displayed, projected, ledger-recorded, and scored value.

During shadow rollout, `probability` remains the market-blended value. After the activation gate passes, set `probability = calibratedProbability` and retain the earlier fields plus map version and `probabilitySource` for audit.

## Implementation

1. Add a pure monotone fitter in `scripts/_forecast-calibration.mjs`, using individual published-origin outcomes from the rolling window and injected timestamps.
2. Fit per domain only when its preregistered minimum sample is met. Sparse domains use identity. A hierarchical/global fallback is out of scope unless a time-separated validation proves non-regression for the receiving domain.
3. Version and persist the map, fit window, cohort filter, per-domain sample, source probability stage, fit timestamp, and code version.
4. Run the new map in shadow. Store both market-blended and calibrated values, but do not change the displayed/scored canonical value.
5. Extend the scorecard with ECE/reliability output and forward raw-vs-calibrated Brier. Use Wilson intervals for realized proportions and entry bootstrap for Brier/ECE uncertainty.
6. Activate only on a later, time-separated cohort. Never evaluate activation on the same outcomes used to fit the map.

## Activation gate

- At least 60 forward published-origin outcomes overall and at least 30 in any domain receiving a non-identity map.
- Paired bootstrap over entry-level Brier loss shows calibrated probability is non-inferior overall and in every activated domain.
- Coverage, VOID, and origin mix are reported beside skill so selection changes cannot masquerade as improvement.
- The map is monotone, versioned, and reversible to identity without losing raw probability lineage.

## Tests

- Deterministic golden fit on a frozen published-origin ledger.
- Monotonicity property test.
- Identity fallback for insufficient total/domain sample, with positive controls for both branches.
- Population filter excludes `bet_engine` and `state_derived`.
- Shadow mode preserves canonical `probability`; activated mode changes it and preserves lineage.
- Fit and evaluation cohorts cannot overlap.
- No live clock in fixtures.

## Acceptance

- [ ] #7071 is live and the fitter consumes the post-blend field.
- [ ] Published-origin-only fit metadata is inspectable.
- [ ] Sparse domains remain identity unless transfer is validated.
- [ ] Shadow raw/blended/calibrated scores are reported on a time-separated cohort.
- [ ] Activation gate passes before canonical `probability` changes.
- [ ] Every consumer of canonical probability, including projections and the resolution ledger, is tested after activation.
- [ ] Rollback to identity is documented and tested.

Contributor guide

Open the contributing guide

Research direction

Read #7071 and the post-market-blend input path first, then inspect scripts/_forecast-calibration.mjs and the probability lineage consumers. Use the listed deterministic fit, monotonicity, fallback, population-filter, shadow/activation, and cohort-separation tests as the implementation checkpoints. Done means a versioned published-origin map is evaluated out of sample and activation, lineage, reporting, and identity rollback are tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, typescript
Domain
analytics, backend, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.