koala73 / koala73/worldmonitor
fix(forecast): judged questions + state-derived titles too vague to resolve — extend sharp questions beyond conflict/political (#5087)
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
## fix(forecast): judged questions + state-derived titles are too vague to resolve (VOID risk)
Companion to the funnel issue. Two generation-side quality gaps make judged forecasts hard for the LLM judge (#5087) to resolve, inflating VOID/disagreement.
### 1. Generic judged questions outside conflict/political
`buildQuestion` (`scripts/_forecast-resolution.mjs`) only emits a sharp, escalation-framed question for `domain === 'conflict'` (#5136) and `domain === 'political'` (#5091). Every other judged domain gets the generic template:
> `Will "" (<domain>, <region>) resolve YES within its <horizon> horizon?`
For cyber / supply_chain / market / military this is ambiguous — "resolve YES" has no crisp criterion — so the judge disagrees or VOIDs. The Jul-14 maturing cohort is exactly these (cyber + supply_chain), so this bites first.
- **Fix:** extend the family-aware, criterion-explicit question to cyber ("did a materially elevated volume of cyber incidents affecting <region> occur…"), supply_chain ("did a material disruption at <route/chokepoint>…"), market ("did <instrument/metric> move materially versus its baseline…"), military. Keep the generic as last-resort only.
### 2. `state_derived` auto-titles are unresolvable prose
State-derived forecasts carry machine-built titles like *"Maritime energy flow disruption from Black Sea maritime disruption state (market)"* — even a human can't state the YES criterion. Since the judged `question` interpolates the title, a bad title = a bad question.
- **Fix:** give `buildStateDerivedForecast` a clean title/claim template (subject + region + measurable condition + horizon), or resolve state-derived forecasts from their structured fields (bucket/region/threshold) rather than the display title.
### Validation
Watch judged `voidRate` per domain on the scorecard after deploy; a drop on cyber/supply/market confirms the fix. RED-first tests: `buildQuestion` emits a domain-specific, non-generic question for each judged domain.
Related: #5087 (judged resolver), #5136/#5091 (the conflict/political question templates to mirror), and the funnel issue (title quality is part of the same input-quality problem).
Contributor guide
Research direction
Start in scripts/_forecast-resolution.mjs by reading buildQuestion and the existing conflict and political branches referenced by #5136 and #5091, then inspect buildStateDerivedForecast. Add RED-first coverage for domain-specific, non-generic questions across the listed judged domains and ensure state-derived forecasts have resolvable claims; validate the result through per-domain voidRate on the scorecard after deployment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, typescript
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100