koala73 / koala73/worldmonitor
feat(forecast): market blend — shift published weight toward the anchor and measure deviation skill
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
Part of #7066. This issue changes the market blend before #7070 fits an empirical calibration map.
## Goal
Move the published forecast closer to a liquid market anchor as a provisional risk-control change, preserve the internal forecast as an auditable disagreement signal, and measure whether the internal deviation ever earns weight back.
## Corrected evidence
- Current weight: `0.4 * market + 0.6 * internal`, before domain caps.
- Pooled scorecard: WM Brier 0.149 vs market 0.118, n=41.
- The pooled n=41 is not one published-product population: it contains 26 published-origin, 13 bet-engine, and 2 state-derived entries.
- The bet-engine n=13 cohort is a subset of the pooled n=41, not independent confirmation.
- Published-origin matched cohort: current WM Brier about 0.0957 vs market 0.0640, n=26.
- A stored-drift counterfactual at weight 0.9 is about 0.0646 on the published matched cohort before domain caps. This supports a provisional 0.9 test; it is not a mathematical guarantee.
- Existing `deviationSkill` is a thin bet-engine statistic. It does not justify deviation by the published generator.
## Scope
### 1. Provisional blend
Raise `MARKET_CALIBRATION_WEIGHT` from 0.4 to 0.9 where the existing liquid-anchor matching, price validity, volume floor, and domain caps pass.
Before code activation, run a fixed-cohort counterfactual over the exact 26 published anchored windows for weights 0.4, 0.7, 0.9, and 1.0. Record paired entry losses and domain composition. Keep 0.9 as the intended rollout value unless this replay reveals a code/cap discrepancy.
### 2. Preserve probability lineage
Before overwriting canonical probability, persist:
- `internalProbability`: pre-market value;
- `marketBlendedProbability`: post-blend/post-cap value;
- existing `calibration.marketPrice`;
- existing `calibration.drift` as `internalProbability - marketPrice`.
Do not add a second `marketDrift` field with the same meaning. If a clearer name is required, perform an explicit compatible migration and update all consumers.
### 3. Published-origin deviation measurement
Add a separate published-origin statistic using the unblended `internalProbability` against the same market outcome:
- fixed disagreement band and window, preregistered before new outcomes accrue;
- paired Brier-loss difference and entry-level bootstrap interval;
- per-domain sample and result;
- no use of the 0.9-blended value to select deviation cases.
The existing bet-engine `deviationSkill` remains a shadow-engine measure and is not the earn-back gate for the published blend.
### 4. Defer automatic earn-back
Automatic per-domain weight changes are not part of this issue. A later issue may define them only after a domain has at least 30 preregistered published-origin deviation cases and a positive paired-skill interval. That future contract must state the exact resulting weight and rollback rule.
## Verification
- Unit tests for 0.9 blend, domain caps, invalid/low-volume anchors, and no-match behavior.
- Probability lineage fields round-trip through history, ledger, dashboard, case files, and generated API surfaces.
- Counterfactual fixture reproduces the current and proposed paired losses.
- Published-origin deviation metric excludes shadow/synthetic origins and uses the unblended value.
- Documentation describes 0.9 as provisional and states that lower Brier is better.
## Rollout and follow-up
1. Record the fixed counterfactual cohort and result.
2. Deploy 0.9 with a reversible constant/config change.
3. Monitor matched published-origin outcomes until at least 30 new matches or 30 days, whichever is later.
4. Report paired WM-minus-market Brier difference, interval, domain mix, and coverage.
5. Roll back if the paired loss regression exceeds the preregistered non-inferiority margin or data integrity changes.
## Acceptance
- [ ] Corrected cohort facts replace the independence claim.
- [ ] Fixed-cohort counterfactual is recorded before activation.
- [ ] Weight 0.9 is live behind a reversible change; matching guards and caps remain.
- [ ] Internal, market-blended, market, and drift lineage is published without duplicate semantics.
- [ ] Published-origin deviation measurement is separate from bet-engine `deviationSkill`.
- [ ] No automatic earn-back policy is wired in this issue.
- [ ] Follow-up observation has a fixed cohort, uncertainty method, decision threshold, and rollback rule.
Contributor guide
Research direction
No files or tests are named. Start by locating MARKET_CALIBRATION_WEIGHT and the existing deviationSkill implementation, then trace probability consumers through history, ledger, dashboard, case files, and generated API surfaces. Done means the fixed-cohort replay, reversible 0.9 blend, lineage fields, separate published-origin metric, tests, and documentation satisfy the acceptance checklist without automatic earn-back.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- analytics, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100