Add semantic quality gates for generated threat models

Problem

A generated threat-model artifact can satisfy its JSON schema and reference-integrity checks while still being materially incomplete or unreliable. Structural validation confirms shape, required fields, and valid references; it does not establish that the analysis is sufficiently comprehensive, internally consistent, or supported by the cited source.

Generic quality failures can include:

• incomplete coverage of discovered components, assets, boundaries, interfaces, and lifecycle transitions;
• evidence anchors that identify a relevant file but do not support the exact claim;
• claims that conflict with available source;
• duplicate or substantially overlapping scenarios;
• unknowns that remain unresolved even though the repository contains enough information;
• missing or understated coverage gaps.

Because the artifact drives downstream planning and task routing, these problems can amplify into missed work and wasted effort even when the node reports successful validation.

Proposed improvement

Add a semantic review stage after schema validation and before the artifact is accepted for downstream routing.

The review should:

• derive a coverage matrix from project-discovery and actor/flow handoffs;
• verify that evidence supports the exact mechanism asserted by each modeled scenario;
• detect source-contradicted assertions;
• normalize and deduplicate overlapping scenarios;
• re-check unknowns against available source before retaining them;
• surface semantic quality diagnostics separately from schema validity;
• trigger a focused revision or mark the run as degraded when minimum quality criteria are not met.

Add regression fixtures with curated reference models so changes to prompts, models, or workflow logic can be evaluated for coverage, evidence accuracy, contradiction rate, and duplication.

Acceptance criteria

• Schema validity and semantic quality are reported as separate statuses.
• A schema-valid artifact is not treated as fully accepted when semantic checks fail.
• Every accepted modeled scenario has evidence supporting its exact claim.
• Source-contradicted claims and source-resolvable unknowns are rejected or revised.
• Coverage is reviewed across discovered assets, trust boundaries, attack surfaces, and lifecycle transitions.
• Duplicate or materially overlapping scenarios are flagged before downstream routing.
• Regression evaluation detects meaningful losses in coverage or evidence accuracy.
• Downstream nodes receive either a semantically accepted artifact or an explicit degraded-quality signal.

Start by tracing the existing schema-validation stage, project-discovery and actor/flow handoffs, and the downstream routing nodes. Define how semantic status, coverage, evidence, contradiction, unknown, and duplication checks fit into that workflow, then add curated reference-model fixtures and verify degraded-quality signaling and regression evaluation.