Encode APG patterns as reusable ARIA conformance contracts (internal aria-spec suite)
- Dominant language
- TypeScript
- Stars
- 13k
- Forks
- 1.1k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 690
Description
## Summary
We've landed a lot of accessibility work recently, but the knowledge lives in ~120 per-component tests with no shared contract. There's no way to answer *"which APG patterns are fully covered, across which components?"* This proposes an **internal, unpublished** package (`@astryxdesign/aria-spec` at `internal/aria-spec/`) that encodes each [WAI-ARIA APG](https://www.w3.org/WAI/ARIA/apg/patterns/) pattern as a **reusable, data-driven conformance contract**, and binds components to it. It covers **behavior, structure, and (scoped) visual** checks, and emits a **coverage matrix** so conformance becomes a tracked metric rather than scattered green/red tests.
Internal-only to start (`private: true`, alongside `test-utils`, `vibe-tests`, etc.). Not published to npm.
## Motivation
- **No pattern-level view.** APG interaction IDs already show up ad-hoc in test names (`menus-11`, `menus-13`, `checkbox-42`, `switch-1`) — the instinct exists, but there's no system behind it.
- **Contracts are copy-pasted.** Each component re-derives "a checkbox should toggle on Space, expose `aria-checked`, be labelled." That logic should be authored once per pattern and reused.
- **Gaps are invisible.** When a component partially conforms, that gap isn't recorded anywhere greppable.
- **Unused deps.** `@axe-core/playwright` and `@playwright/test` are in `package.json` but wired to nothing.
## Design
### Three-layer architecture
```
internal/aria-spec/ (private, unpublished)
├── patterns/ one contract per APG pattern, authored as DATA + assertions
│ switch.ts → expectations: [
│ { id: 'switch-role-and-checked', priority: 'blocker', run },
│ { id: 'switch-space-toggles', priority: 'blocker', run },
│ { id: 'switch-labelled', priority: 'blocker', run },
│ { id: 'switch-optionally-described', priority: 'major', run },
│ ...
│ ]
├── primitives/ keyboard / focus / aria / dom helpers (runtime-agnostic)
├── harness/ AriaHarness adapter interface (jsdom + browser implementations)
├── runner/ runContract(pattern, { harness, bindings, expectedFailures })
└── report/ emits JSON → coverage matrix (pattern × component × expectation)
```
### 1. Patterns as data, with priority tiers
Each pattern is a typed set of **expectations**, each tagged with a priority: `blocker | major | minor | optional`. CI gates on `blocker`; the rest are reported. This lets us gate meaningfully without blocking on optional-authoring niceties.
### 2. Components bind the contract + declare `expectedFailures`
The per-component test stays tiny and is **honest about gaps**:
```ts
runSwitchContract({
render: () => {}} />,
expectedFailures: [
// tracked gap, not a missing test
],
});
```
`expectedFailures` is the key adoption mechanism: we roll the contract across all 122 components immediately, and **every gap becomes an explicit, greppable, reviewable line** instead of a missing test. Burning down `expectedFailures` becomes the hardening backlog.
### 3. Coverage matrix
The runner emits `pattern × component × expectation` results → a coverage report (% of expectations passing per component, per pattern). This turns APG conformance into a dashboard metric.
## Runtime: two tiers, runtime-agnostic contracts
Expectations are authored against an abstract **`AriaHarness`** adapter (render / query-by-role / get-attribute / press-key / focus / accessible-name), so the *same* contract runs in either tier. This mirrors the proven internal ARIA-spec design (specs take an optional `device` param; default = in-process, override = real browser).
### Tier 1 — jsdom (default, fast, every PR)
Vitest + Testing Library + `user-event` — the stack our existing tests already use. Genuinely covers the majority of APG expectations: role presence, accessible-name computation, ARIA state wiring (`aria-expanded`/`-controls`/`-checked`/`-selected`/`-activedescendant`, `aria-labelledby`/`-describedby`), and keyboard behavior (arrows, Home/End, typeahead, Enter/Space, Escape, Tab moving `activeElement`). Runs in the existing `pnpm test` path.
### Tier 2 — real browser (gated, for fidelity-critical expectations)
jsdom **fakes** exactly the things composite APG widgets depend on, so a browser tier is necessary — not optional — for full conformance:
- The **real accessibility tree** as exposed to AT (jsdom only approximates it) → `toMatchAriaSnapshot`
- Focus-trap Tab **wrapping**, `inert`, native `.showModal()`, Popover API **top layer** (all currently mocked in our tests — e.g. Switch/DropdownMenu stub `showPopover`)
- `:focus-visible` and forced-colors state (CSS-engine, invisible to jsdom)
**Runner decision: Vitest Browser Mode** (`@vitest/browser` + Playwright/Chromium provider), added as a **third Vitest project** (`browser`) alongside the existing `ui`/`node` projects. Rationale: one runner, one config, same `render()`/Testing-Library API as Tier 1 — but `document` is real Chromium, and the Playwright provider gives native `toMatchAriaSnapshot` + `toHaveScreenshot`. Only fidelity-critical expectations opt into this tier.
Runner-up considered: **Storybook Test Runner** (reuses our 169 stories as fixtures, and CI already builds Storybook) — attractive for a cheap axe-per-story sweep, but it's a second test system with its own conventions. Likely a *complement* later (axe-per-story in the storybook job), not the primary contract runtime.
### Structure & visual layers
- **Structure (static):** wire **axe-core** (`vitest-axe` in Tier 1) as a per-component pass — contrast/name/dup-id/role violations. Complements, does not replace, the behavior expectations.
- **Structure (tree):** accessibility-tree snapshot (`toMatchAriaSnapshot`) — a *text* YAML of roles/names, reviewable in PRs, no pixels. Tier 2.
- **Visual (pixels):** scoped tight — screenshots **only** for inherently-visual a11y states (`:focus-visible` ring, forced-colors/high-contrast, reduced-motion). Real screenshots + browser, its **own opt-in/nightly job**, ~2 themes. Not in the blocking path.
## CI shape
Leverages the existing `check-scope` gating (jobs already skip docsite-only PRs) and the existing `build-storybook` job.
```
check-scope ──┬─► test (jsdom Tier 1, fast) ← blocker + all, every PR [existing]
├─► build-storybook ← + optional axe-per-story sweep
├─► aria-spec-browser (Tier 2) ← blocker-gated, component PRs only, cached Chromium
└─► aria-spec-visual (screenshots) ← nightly / labeled, non-blocking
```
Streamlining levers:
- **Gate Tier 2 on a `component_changed` scope output** (extend the existing changed-files step) — don't run the browser tier on non-component PRs.
- **Gate the build only on `blocker` expectations**; upload the full coverage matrix as an artifact for `major`/`minor`. Lets us enable across 122 components without a wall of red.
- **Cache the Playwright browser** (`~/.cache/ms-playwright`, keyed on Playwright version) — the install is the slow part, not the tests.
- **Chromium-only per-PR** (the a11y tree + keyboard are engine-consistent); cross-browser is a nightly concern.
- **Shard Tier 2** (`--shard=i/N`) only if/when the suite grows.
## Consolidation
Existing a11y assertions scattered across `*.test.tsx` should **move/consolidate into the pattern contracts**. The ad-hoc APG-tagged tests (`menus-*`, `checkbox-42`, `switch-1`) become the first `expectedFailures`-free contract bindings. Goal: one authoritative place per pattern, per-component tests reduced to a binding + a gap list.
## Scope of the first prototype (separate PR)
Prove the architecture end-to-end on **one pattern** before authoring all patterns:
- [ ] `internal/aria-spec/` package scaffold (`private: true`), `AriaHarness` interface + jsdom adapter
- [ ] Runner with priority tiers + `expectedFailures`, coverage-matrix emit
- [ ] Author the **Switch/Checkbox** pattern contract (existing `switch-1`/`checkbox-42` tags to migrate)
- [ ] Bind Switch (+ Checkbox) as jsdom (Tier 1) contract tests, running in the existing suite
- [ ] Wire the **Tier 2 browser project** (Vitest Browser Mode) + one browser binding proving `toMatchAriaSnapshot`
- [ ] Emit a JSON coverage matrix + human-readable summary
- [ ] Migrate the existing ad-hoc APG-tagged assertions into the new contract
## Explicitly out of scope (follow-ups)
- `vitest-axe` structural pass wiring (dep not yet installed)
- Pixel screenshots for focus/contrast states
- Authoring the remaining ~22 patterns
- Coverage dashboard UI
- Storybook axe-per-story sweep
## Open questions
- Package name / location: `@astryxdesign/aria-spec` at `internal/aria-spec/` — OK?
- Pattern authoring format — pure TS modules vs. a declarative spec object the runner interprets?
- Do we gate CI on `blocker` expectations from day one, or report-only until coverage is broad?
Contributor guide
Assessment
This issue has not been assessed yet.