koala73 / koala73/worldmonitor

fix(forecast): judged questions + state-derived titles too vague to resolve — extend sharp questions beyond conflict/political (#5087)

Open
#5,234 1 comment 0 reactions 0 assignees View on GitHub
area: AI/intel bug
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## fix(forecast): judged questions + state-derived titles are too vague to resolve (VOID risk)

Companion to the funnel issue. Two generation-side quality gaps make judged forecasts hard for the LLM judge (#5087) to resolve, inflating VOID/disagreement.

### 1. Generic judged questions outside conflict/political
`buildQuestion` (`scripts/_forecast-resolution.mjs`) only emits a sharp, escalation-framed question for `domain === 'conflict'` (#5136) and `domain === 'political'` (#5091). Every other judged domain gets the generic template:
> `Will "" (<domain>, <region>) resolve YES within its <horizon> horizon?`
For cyber / supply_chain / market / military this is ambiguous — "resolve YES" has no crisp criterion — so the judge disagrees or VOIDs. The Jul-14 maturing cohort is exactly these (cyber + supply_chain), so this bites first.
- **Fix:** extend the family-aware, criterion-explicit question to cyber ("did a materially elevated volume of cyber incidents affecting <region> occur…"), supply_chain ("did a material disruption at <route/chokepoint>…"), market ("did <instrument/metric> move materially versus its baseline…"), military. Keep the generic as last-resort only.

### 2. `state_derived` auto-titles are unresolvable prose
State-derived forecasts carry machine-built titles like *"Maritime energy flow disruption from Black Sea maritime disruption state (market)"* — even a human can't state the YES criterion. Since the judged `question` interpolates the title, a bad title = a bad question.
- **Fix:** give `buildStateDerivedForecast` a clean title/claim template (subject + region + measurable condition + horizon), or resolve state-derived forecasts from their structured fields (bucket/region/threshold) rather than the display title.

### Validation
Watch judged `voidRate` per domain on the scorecard after deploy; a drop on cyber/supply/market confirms the fix. RED-first tests: `buildQuestion` emits a domain-specific, non-generic question for each judged domain.

Related: #5087 (judged resolver), #5136/#5091 (the conflict/political question templates to mirror), and the funnel issue (title quality is part of the same input-quality problem).

Contributor guide

Open the contributing guide

Research direction

Start in scripts/_forecast-resolution.mjs by reading buildQuestion and the existing conflict and political branches referenced by #5136 and #5091, then inspect buildStateDerivedForecast. Add RED-first coverage for domain-specific, non-generic questions across the listed judged domains and ensure state-derived forecasts have resolvable claims; validate the result through per-domain voidRate on the scorecard after deployment.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, typescript
Domain
ai
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.