Automattic / Automattic/data-liberation-agent
The comparison gate: half of it bypasses the check registry, and it writes no report
- Dominant language
- TypeScript
- Stars
- 31
- Forks
- 3
- Avg merge
- 10h 14m
- Merged PRs (30d)
- 81
Description
`src/lib/fidelity/checks.ts` introduced a proper extension point for the comparison, and its own doc comment states the principle:
> The built-in comparison registers through this same call rather than a privileged branch, because a door only one caller can open is not a door.
That principle holds for one of the three things `compare` actually does. The other two still go through privileged branches, and the run leaves nothing behind that a later run could be measured against.
## 1. The tier that scales is not extensible
`checkFidelity` runs two tiers:
- **Self-consistency** — dangling links, unresolved anchors, remote assets. Pure functions of what is on disk, so it runs over **every** route.
- **Source fidelity** — browser observation against the live source. Costs a round trip, so it runs over a bounded sample.
Only the second goes through the registry. `checkSelfConsistency` is a direct call in `check.ts`, so a contributed check can never participate in the whole-site tier — it is limited to the sampled routes. The cheap, exhaustive, most useful tier is the closed one.
## 2. The interactivity pass fabricates observations
The 390px interactivity pass builds a `LayoutObservation` with every field blanked except `dialogs`:
```ts
const dialogOnly = ( observation: LayoutObservation ): LayoutObservation => ( {
...observation,
title: 'x',
textChars: 0,
widestImage: null,
images: [],
overflow: false,
docWidth: 390,
externalHosts: [],
hashTargets: [],
internalMissing: [],
} );
```
and feeds it to `scoreViewport`, bypassing the registry. The reasoning — that handing a contributed check a synthetic observation would invite conclusions drawn from zeroed values — is correct. The problem is the type: `LayoutObservation` now means "measured" on one path and "mostly fabricated" on another, with nothing distinguishing them. `title: 'x'` exists purely to defeat an equality check.
A narrow question deserves a narrow type. An `InteractivityObservation` (or a check that declares which fields it reads) removes the need to lie.
## 3. `compare` writes no report
Every other stage emits a schema-versioned sidecar: `capture-receipt.json`, `artifact.json`, `semantic-evidence.json`, `layout-geometry-report.json`, `interaction-states.json`. `compare` writes PNGs and prints to stdout. `FidelityReport` is returned in-process and, over MCP, serialised into a tool result.
Nothing lands on disk, so there is no baseline, no regression detection, no way to diff two runs of the same site, and no artifact a reviewer can be pointed at. For a gate whose whole job is to decide whether a copy is faithful, the verdict is the one thing not preserved.
## 4. The gate's own scope controls are unreachable
`FidelityCheckOptions` declares `routes` and `sampleSize`. `src/cli.ts` passes only `screenshots`, and `src/ui/compare.ts` forwards only `directory`, `screenshots`, and `log`. The default gate therefore compares four routes at two widths, with no way to widen it short of writing code.
The output is honest about this — `ui/compare.ts` prints the measured scope rather than a bare "passed" — but honesty about an unchangeable default is not the same as control.
## 5. Thresholds have no policy
Tolerances are scattered constants with no shared rationale: `textChars` must match exactly, `IMAGE_TOLERANCE_PX = 2`, `MISSING_IMAGE_TOLERANCE = 1`, extra images never fail, and `rendered-contract-checks.ts` adds `IMAGE_POSITION_TOLERANCE_PX`, `IMAGE_SIZE_TOLERANCE_PX`, `TYPOGRAPHY_METRIC_TOLERANCE_PX`, `TYPOGRAPHY_COVERAGE_FLOOR`, `MOTION_COVERAGE_FLOOR`. Each is individually justified in a comment; together they are an unstated definition of "perfect" that no caller can inspect or tune.
## Proposed change
1. Register self-consistency as a whole-site check, and let the registry distinguish per-route checks from whole-site ones.
2. Give the interactivity pass its own observation type instead of a blanked `LayoutObservation`.
3. Write a schema-versioned `compare-report.json` next to the receipt, containing the per-route, per-viewport results, the checks that ran, the thresholds in force, and the evidence paths.
4. Expose `--routes` and `--sample` on `data-liberation compare`.
5. Collect the thresholds into one declared policy object reported in the artifact.
## Acceptance
- A contributed check can run over every route.
- No code path constructs a `LayoutObservation` it did not measure.
- Two `compare` runs of the same site can be diffed from disk.
- `compare --routes /,/shop/` works.
---
*AI assistance: researched and written by Claude via Claude Code, from reading `check.ts`, `checks.ts`, `score.ts`, `self-consistency.ts`, `rendered-contract-checks.ts`, and the CLI wiring. Chris Huber orchestrated and reviewed the work and is responsible for what is filed here.*
Contributor guide
Research direction
Start by reading src/lib/fidelity/check.ts, checks.ts, score.ts, self-consistency.ts, rendered-contract-checks.ts, and the CLI wiring in src/cli.ts and src/ui/compare.ts. Trace how checks, observations, options, thresholds, and reports currently flow before choosing an implementation approach. Done means the four acceptance criteria pass, including compare --routes /,/shop/ and diffable compare-report.json output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cli, testing-qa, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100