Track typed acceptance criteria through the review and repair loop

The problem

One issue I’ve repeatedly hit in my own workflows is agents declaring work complete without fully delivering what I intended. Even after an interview, acceptance criteria and preparing UX and architecture, I’d wake up to “work done” and still have to point out obvious gaps.

I’ve noticed this most with larger features. I already have this working in my own Roadmap plugin: AI handles larger scopes with less supervision, using the captured intent, UX and architecture to review the result and keep working on the gaps. That experience is what I’m bringing to this suggestion.

Assumption to validate

Have you encountered this with Facility, or does the current story, review and repair workflow already catch these gaps?

What you propose

Facility already has a review-and-repair loop. I’d extend it to track each original acceptance criterion and keep working on the remaining gaps within agreed scope and budget, until every criterion is verified or a blocker or limit is reported. Typed review results are the first step.

Roadmap taught me the value of validating relationships between original criteria, review results and completion. This proposal starts with typed review results inside Facility’s existing workflow, using its TypeScript/Zod contracts, reviewer template and run evidence.

Workflow

flowchart TD
    request["Issue / story<br/>Intent, UX and initial criteria"] --> plan
    subgraph plan["Architect: plan"]
        planned["PR2: Type the agreed criteria<br/>Validate IDs and descriptions"]
    end
    plan --> builder[Builder implements and checks]
    builder --> review
    subgraph review["PR reviewer: check criteria"]
        direction TB
        reported["PR1: Type review results<br/>Status and evidence per criterion"]
        covered["PR2: Check results against<br/>the full typed criterion list"]
        reported ~~~ covered
    end
    review -->|GitHub findings| repair
    subgraph repair["Address-review: repair"]
        remaining["PR3: Use criterion status to continue<br/>or stop at a blocker / scope / budget limit"]
    end
    repair -->|PR updated| review
    classDef first fill:#ddf4ff,stroke:#0969da,color:#0550ae,stroke-width:2px
    classDef later fill:#ffffff,stroke:#0969da,color:#0550ae,stroke-width:2px,stroke-dasharray:3 4
    class reported first
    class planned,covered,remaining later
    style plan fill:#f6f8fa,stroke:#8c959f
    style review fill:#f6f8fa,stroke:#8c959f
    style repair fill:#f6f8fa,stroke:#8c959f

• Outer boxes and arrows: Facility’s existing workflow.
• Solid blue: PR1, typed review results.
• Dotted blue: proposed PR2 and PR3, extending the same steps.

What becomes typed

Existing record	Proposed extension
Agreed criteria in conversation and planning files	PR2: Capture that agreed list as Zod-validated data with stable IDs and descriptions.
Review findings and evidence	PR1: Validate a status and evidence per criterion. PR2: Check those results against the full agreed list.
Review threads and repair summaries	PR3: Use criterion status to continue the existing loop within scope and budget, or report the blocker and remaining gaps.

PR2 structures criteria Facility already records; it does not introduce another planning step. Zod validates each record’s shape, and a coverage check compares criterion IDs across the agreed list and review results.

Suggested implementation of PR1

flowchart TD
    prompt["pr-reviewer.md<br/>Extend output instructions"] --> dispatcher["TurnDispatcher + acceptanceReviewEvent<br/>Parse and redact final report"]
    contract["@facility/agents<br/>Proposed AcceptanceReviewReportSchema"] -.->|Validate| dispatcher
    dispatcher --> events["appendTurnEvent / turnEvents<br/>Reuse existing storage"]
    events --> projection["presentTurnEvent<br/>Extend criterion presentation"]
    projection --> ui["RunActivity<br/>Reuse existing UI"]
    classDef changed fill:#ddf4ff,stroke:#0969da,color:#0550ae
    class prompt,dispatcher,contract,projection changed

Blue marks the proposed additions and extensions. Storage and the UI component are reused. The dispatcher reads the completed final response, validates the report, redacts its text and appends an event linked to the run. The existing activity view presents the criterion results.

Suggested review-report contract

Zod rejects missing required fields and invalid statuses at runtime. Its inferred TypeScript type catches shape errors in typed code. The report could live in @facility/agents.

import { z } from "zod";

export const AcceptanceReviewReportSchema = z.object({
  schemaVersion: z.literal(1),
  results: z.array(z.object({
    criterionId: z.string().trim().min(1).max(80),
    criterionSnapshot: z.string().trim().min(1).max(512),
    status: z.enum(["met", "unmet", "blocked", "unverified"]),
    evidence: z.string().trim().min(1).max(1_024),
  }).strict()).min(1).max(20),
}).strict().refine(
  report => new Set(report.results.map(r => r.criterionId)).size
    === report.results.length,
  { message: "Criterion IDs must be unique" },
);

export type AcceptanceReviewReport =
  z.infer<typeof AcceptanceReviewReportSchema>;

criterionSnapshot records the wording reviewed. The numeric limits are provisional.

Statuses: met = verified; unmet = gap found; blocked = verification prevented; unverified = not checked.

This validates submitted reports. It does not require a report, detect omitted story criteria or prove the evidence.

Example: agreed UX versus delivered feature

{
  "schemaVersion": 1,
  "results": [
    {
      "criterionId": "AC1",
      "criterionSnapshot": "The user can complete the agreed workflow through the UI.",
      "status": "met",
      "evidence": "Completed the flow through the UI and verified the result."
    },
    {
      "criterionId": "AC2",
      "criterionSnapshot": "Implement the approved screens, interactions and UI states.",
      "status": "unmet",
      "evidence": "Compared the implementation with the approved UX. The error state and retry interaction are missing."
    }
  ]
}

Expected behavior

• Validate reports from the final response and show each result with its evidence in run activity.
• Show a diagnostic for malformed reports; keep unstructured runs working.
• Reuse existing run identity, revision evidence, redaction, access checks and review/repair triggers.

No new table, API, acceptance state or scheduler.

What you considered instead

• Keep the existing workflow: Facility already asks reviewers to check acceptance criteria and supports repair. This may be sufficient in practice; I’d first validate whether criteria get missed or gaps remain unresolved.
• Strengthen prompts and use a consistent checklist: Ask reviewers to account for every criterion and carry unresolved items into repair. This could address much of the problem with less code. Typed results would make those outcomes easier to compare and use consistently across runs.
• Add an independent completion judge: Check the delivered feature against the original intent, UX and architecture. This directly addresses the gaps I’ve encountered, but adds another review pass. I’d first explore strengthening the existing reviewer’s role.
• Implement the full completion loop immediately: Require coverage of the agreed criteria and continue repairs within scope and budget. I’d start with typed review results, then agree how those results should control continuation and completion.

Start with pr-reviewer.md and the TurnDispatcher/acceptanceReviewEvent entry points, then inspect the existing @facility/agents contracts and appendTurnEvent, turnEvents, presentTurnEvent, and RunActivity flow. Trace how final responses are parsed, redacted, stored, and displayed. Done means malformed reports produce diagnostics, valid criterion results appear in run activity, and unstructured runs continue to work.