Gate 2 governance: blast-radius scoring, test mutation validation, and multi-reviewer consensus on critical surfaces

Open
#200 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
28/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
typescript

Research direction

Start with docs/concepts/method.md for the existing Gate 2 model, then inspect packages/harness/src/blast-radius.ts and the named critical-surface paths. Define how blast-radius scoring, mutation validation, and dual-reviewer approval connect to delivery state transitions. Done means high-risk changes are escalated, weak tests are flagged, and critical surfaces require the proposed human consensus.

Written by the indexing model from the issue text.

Description

Problem

Facility's operating model places absolute trust in Two Inviolable Human Gates (docs/concepts/method.md):

  1. Gate 1: Human judges and accepts the /architect plan.
  2. Gate 2: Human validates the preview and performs the squash-merge. "Every merge carries a human decision."

While Gate 2 successfully prevents runaway autonomous pushes, scaling agent delivery reveals a fundamental governance gap: Gate 2 treats all PRs as equal-risk and assumes green CI checks equal meaningful test assertions.

What breaks at scale

  1. Single-reviewer alert fatigue: When an agent swarm delivers 10+ PRs daily, a single reviewer inevitably skims diffs and leans on green CI checks as proof of correctness.
  2. Tautological agent tests: LLMs frequently write test suites that achieve 100% line coverage while asserting weak or self-fulfilling invariants. A flawed implementation accompanied by a flawed test suite passes all CI checks cleanly.
  3. Flat risk policy: A documentation typo fix and a PR modifying services/gateway/src/auth.ts, guards/, or packages/db/src/schema.ts pass through the exact same single-maintainer squash-merge flow.

If a single reviewer is fatigued, rushed, or compromised, green CI status allows subtle regressions or security backdoors on critical surfaces to merge undetected.

Proposed architecture

We propose hardening Gate 2 with three tiered governance mechanisms:

                      [Agent PR Created]
                              │
                              ▼
            ┌───────────────────────────────────┐
            │  1. Diff Blast-Radius Scorer      │
            └───────────────────────────────────┘
               ├── Low Risk (docs / UI) ─────► Standard Gate 2 (Single Human)
               └── High Risk (auth / db / guards) ──┐
                                                    ▼
                                   ┌───────────────────────────────────┐
                                   │  2. Adversarial Mutation Runner   │
                                   └───────────────────────────────────┘
                                                    │
                                                    ▼
                                   ┌───────────────────────────────────┐
                                   │  3. Dual-Reviewer Consensus Gate  │
                                   └───────────────────────────────────┘
1. Diff Blast-Radius Scoring (packages/harness/src/blast-radius.ts)

Introduce deterministic risk scoring for every delivery branch based on modified paths, entropy, and sensitive surfaces:

  • Critical surfaces: guards/, .github/workflows/, packages/db/src/schema.ts, services/gateway/src/auth.ts, services/api/src/idempotency.ts, STANDARD.md.
  • Medium surfaces: Backend service endpoints, state machines, SDK contracts.
  • Low surfaces: UI styles, documentation, static assets.
  • If a delivery touches a Critical Surface, the platform flags the story as high_risk and escalates review requirements.
2. Automated Adversarial Mutation Validation

To ensure agent-generated tests actually assert business invariants rather than green execution paths:

  • When a PR adds or modifies tests, run a lightweight AST mutation step in the sandbox (e.g. inverting conditionals, flipping boolean returns, stripping guard clauses).
  • The Ratchet Rule: If the agent's new test suite still passes against intentional mutations, the run fails with suspicious_weak_tests and requires the agent to add assertions that catch the mutants before opening/updating the PR.
3. Multi-Reviewer Consensus on Critical Surfaces
  • Escalated high_risk stories require approvals from two distinct human reviewers (or a designated security owner) before the story can transition to Done or merge into main.
  • Prohibit single-maintainer squash-merges on changes modifying cryptographic operations, virtual key routing, or guard policies.

Expected impact

  • Zero silent regressions on core infrastructure: Critical surfaces gain defense-in-depth protection against single-point-of-failure human reviews.
  • Auditable test integrity: Verifies that agent tests catch real regressions rather than providing false confidence.
  • Sustainable reviewer throughput: Low-risk changes continue shipping fast with single-human approval, focusing human attention strictly where the blast radius is large.
Dominant language
TypeScript
Stars
71
Forks
64
Avg merge
15h 38m
Merged PRs (30d)
66

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from theam/facility

All issues in theam/facility

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.