koala73 / koala73/worldmonitor

feat(intel): corroboration and disagreement layer — build the checkable half of the fact-checking request

Open
#6,419 2 comments 0 reactions 0 assignees View on GitHub
area: AI/intel community design feature P2
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## Request

A user asked for this:

> "Build a fact checking feature that's as close to unbiased as possible. Not selectively enforced. Every detail, from any source or voice, checked regardless of political affiliation."
> "This could fight FUD, Fake News, and/or Governmental Censorship."

## What we have today (verified at `03e4c1e83`)

More of the substrate exists than the request assumes, and the missing piece is named explicitly in our own code.

**Source authority tiers.** `server/_shared/source-tiers.ts` loads `shared/source-tiers.json` into a four-tier ranking, defaulting unknown sources to tier 4:

```
Tier 1: Wire services / official gov/intl orgs — fastest, most authoritative
Tier 2: Major established outlets — high-quality journalism
Tier 3: Specialty / regional / think tank sources — domain expertise
Tier 4: Aggregators and blogs — useful but less authoritative
```

This ranks *authority*.

**Per-feed political lean also already exists — and nothing reads it.** `SourceRiskProfile` declares `knownBiases?: string[]` (`shared/source-provenance.ts:166`), and 15 sources carry a value: `Pro-Ukraine` on nine Ukrainian outlets (`:265`, `:269-271`, `:273-277`), `Anti-Kremlin` on Moscow Times (`:279`) and Meduza (`:281`), `Center-left` on Guardian World (`:309`), `Israeli centre-right` on Jerusalem Post (`:284`), `Israeli mainstream` on Ynetnews (`:285`), `Pro-EU` on EuroNews (`:261`).

A denser second field, `stateAffiliated`, covers 38 sources across 20 states, and it is labelled symmetrically in both directions — `'Voice of America': { risk: 'medium', stateAffiliated: 'USA' }` (`:264`) sits beside `'RT': { risk: 'high', stateAffiliated: 'Russia' }` (`:180`).

`knownBiases` has **zero consumers**. `describePropagandaBadge` (`:368-401`) uses risk, note, and `stateAffiliated` only; `getSourceProvenanceState` (`:415-427`) returns eight fields and omits it; the MCP schema at `api/mcp/registry/nlp-tools.ts:76-81` omits it too. The data is curated and never surfaced.

That matters for scoping: the symmetry the user asked for — "every detail, from any source or voice, regardless of political affiliation" — is already the labelling convention. Surfacing an existing curated field is far smaller than the layer originally scoped below.

Coverage is thin, though, and sizing should reflect it: 66 of 582 sources have a reviewed risk, 204 have a reviewed type, and 516 sit at `risk: 'unknown'`.

**Grounding checks on generated text.** `shared/brief-llm-core.js` mechanically rejects LLM summaries that invent proper nouns not present in the source headline, and extracts numeric and date facts for a presence check. The module states its own boundary at `shared/brief-llm-core.js:313-316`:

> Out of scope: Source-level fact-checking. If the source headline already contains the wrong name (wire-service typo), this validator OKs the summary and the fallback ships the wrong fact verbatim.

So we verify that our AI did not invent things. We do not verify that the source was right.

**Per-claim citations are already enforced mechanically** — more strictly than first assumed. In `scripts/_insights-brief.mjs:294-311`, every lead sentence without an `[n]` marker rejects the whole brief (`LEAD_UNCITED`, `:298-299`); the grounding set is built from *only* the cited stories (`:300`), so a sentence cannot borrow a name or number from an uncited sibling; hallucinated proper nouns and numeric facts reject as `LEAD_PROPER_NOUN` / `LEAD_NUMERIC_FACT` (`:302-306`); model-written brackets are stripped and canonical ones appended so a line cannot cite the wrong source (`:326-329`); and the sources array stays in strict index lockstep by substituting rather than filtering (`:337-346`). The rejection vocabulary is a closed set of 8 literals (`:47-56`), and the user-visible list renders via `src/utils/brief-sources.ts:65-95`.

**A hard corroboration threshold already gates promotion.** `scripts/_clustering.mjs:253-258` requires two sources, or entity corroboration, before a cluster can lead a brief, and `_insights-brief.mjs:251` rejects the brief when nothing qualifies. `tests/brief-contract.test.mjs:277` locks it: an all-single-source day yields `null`.

**But that threshold counts feed labels, not publishers**, so syndication currently passes as independent sourcing. That is a shipped correctness defect and is filed separately as #6428. It is a hard prerequisite for anything below — a "reported by 6 sources" badge built on today's count would mislead precisely in the syndication-heavy cases readers most need it.

**Provenance.** `shared/decision-signal-provenance*.ts` carries corroboration state through the decision-signal pipeline. Note that `shared/decision-signal-provenance-contract.ts:170-175` already declares four states — `single_source`, `multi_source`, `independently_corroborated`, `contradicted` — and the validator accepts all four (`shared/decision-signal-provenance.ts:566-568`), but only the first two are ever emitted (`shared/china-decision-signals.ts:585`, `server/worldmonitor/supply-chain/v1/china-corridor-control-towers.ts:171`, both merely testing `publisherIds.size > 1`). **The "this claim is contradicted" slot already exists in the contract and is empty.** This work can fill it rather than define a new vocabulary.

(An earlier draft of this issue cited `shared/content-attribution.ts` as provenance machinery. That was wrong — that module is UTM / `wm_content_*` marketing attribution and has no role here.)

**Nearest tracked work.** #5994 (`discovery(correlation): evaluate cross-source conflict detection`) already asks whether to preserve and surface conflicting source claims, and its acceptance criteria include defining claim identity, provenance, source confidence, and resolution semantics "without overwriting disagreement".

**A prior attempt was deleted.** Commit `7e23c8280` added `src/services/ai-detection.ts` (194 lines, "Disinformation Detection", Bellingcat heuristics); commit `e34b1ca0a` removed it as dead code. Worth reading both before rebuilding in that direction.

## The honest scoping problem

The request has two halves, and they are not equally tractable.

**Buildable now:** corroboration and disagreement. Whether N independent sources report the same claim, which sources disagree, how the tiers of agreeing and disagreeing sources compare, and how a claim changed over time. These are mechanical, auditable, and apply identically to every source — which is exactly the "not selectively enforced" property the user wants.

**Not a feature we can ship:** a neutral arbiter of truth. Adjudicating contested political claims requires judgments that no automated system, and no LLM, makes neutrally. An "unbiased fact check" verdict badge would ship our model's priors as if they were fact, and would be least reliable precisely on the contested claims the user cares most about. It would also become an obvious target for manipulation.

So this issue proposes building the first half well, and explicitly declining the second half rather than approximating it.

## Proposal

A corroboration layer, not a verdict layer.

**Step 0 — fix the count first (#6428).** Corroboration must mean distinct publishers before any of it is shown to a user.

Then, cheapest first:

1. **Surface `knownBiases` and `stateAffiliated`.** The data is curated, symmetric, and currently invisible. Adding it to `getSourceProvenanceState` (`shared/source-provenance.ts:415-427`) and the existing badge path (`:368-401`) is a small change and delivers the "any source or voice" property directly. Pair it with a visible coverage caveat — 516 of 582 sources are unreviewed, so absence of a label must not read as absence of lean.
2. **Single-source flagging.** Mark a claim carried by one publisher family, or only by tier-4 sources, as uncorroborated. Highest value against FUD, lowest risk, no truth judgment required.
3. **Corroboration display.** For a claim, show how many independent publishers carry it, which they are, and their tiers. "Reported by 6 publishers, including 2 tier-1 wires" is checkable by the reader — once #6428 makes that number true.
4. **Disagreement display.** Where sources conflict, show the conflict instead of resolving it. Emit the `contradicted` state that the contract already defines (`decision-signal-provenance-contract.ts:170-175`) and nothing currently produces. Preserve both claims with attribution — the direction #5994 is exploring.
5. **Claim clustering.** Group reports of the same claim across sources. Depends on entity resolution, which #5981 is already improving.
6. **Change tracking.** When a source revises or retracts, surface that the claim changed.
7. **Uniform application.** The rules run over every source with no exception list. That property must be tested, not asserted.

Everything above tells the reader what the evidence looks like and leaves the conclusion to them. Nothing above requires us to declare a claim true or false.

## Explicitly out of scope

- A true/false verdict badge on contested claims.
- Political-lean or bias scores per outlet. Every published bias rating is itself contested; adopting one would import its politics while looking objective.
- Any ranking change that demotes sources by viewpoint rather than by corroboration or tier.

If a verdict layer is ever wanted, it should be a separate decision with its own review, not a quiet consequence of this work.

## Acceptance criteria

- [ ] #6428 has landed: every corroboration count shown to a user counts publisher families, not feed labels.
- [ ] `knownBiases` and `stateAffiliated` are surfaced, with a visible caveat that most sources are unreviewed.
- [ ] A claim view shows corroborating publisher count, the source names, and their tiers.
- [ ] Claims carried by a single publisher, or only by tier-4 sources, are marked uncorroborated.
- [ ] The `contradicted` corroboration state is emitted where sources disagree, rather than remaining a declared-but-unused contract value.
- [ ] Conflicting claims are shown side by side with attribution, and neither is silently dropped.
- [ ] Source revisions and retractions are surfaced.
- [ ] The corroboration rules are proven to apply uniformly: a test asserts no source, tier, or category is exempt.
- [ ] No user-facing surface asserts that a claim is true or false.
- [ ] The tier data behind any displayed judgment is inspectable by the user.

## Related

- #5994 — discovery(correlation): evaluate cross-source conflict detection. Closest existing work; this issue is its product framing.
- #5981 — epic(correlation): entity-resolution-first correlation improvements. Claim clustering depends on it.
- #6225 — feat(sources): includes EUvsDisinfo, a disinformation-tracking source.

## Notes

Reported by a user in community feedback. Sibling issues from the same feedback are cross-linked in a comment below. This issue deliberately delivers less than the request asked for, and says why; that trade should be reviewed before any implementation starts.

Contributor guide

Open the contributing guide

Research direction

Start by reading #6428, shared/source-provenance.ts, shared/decision-signal-provenance-contract.ts, and the existing brief tests in tests/brief-contract.test.mjs. Trace the provenance and clustering entry points before choosing an implementation slice, then use the acceptance criteria to verify publisher-family counts, surfaced source metadata, disagreement handling, and uniform rules without producing truth verdicts.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, typescript
Domain
backend-api-design, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.