facebook / facebook/astryx

Encode APG patterns as reusable ARIA conformance contracts (internal aria-spec suite)

Open
#4,112 4 comments 0 reactions 1 assignee Claimed by @cixzhang View on GitHub
area:core enhancement needs-scoping type:test
Dominant language
TypeScript
Stars
13k
Forks
1.1k
Avg merge
1d 15h
Merged PRs (30d)
690

Description

## Summary

We've landed a lot of accessibility work recently, but the knowledge lives in ~120 per-component tests with no shared contract. There's no way to answer *"which APG patterns are fully covered, across which components?"* This proposes an **internal, unpublished** package (`@astryxdesign/aria-spec` at `internal/aria-spec/`) that encodes each [WAI-ARIA APG](https://www.w3.org/WAI/ARIA/apg/patterns/) pattern as a **reusable, data-driven conformance contract**, and binds components to it. It covers **behavior, structure, and (scoped) visual** checks, and emits a **coverage matrix** so conformance becomes a tracked metric rather than scattered green/red tests.

Internal-only to start (`private: true`, alongside `test-utils`, `vibe-tests`, etc.). Not published to npm.

## Motivation

- **No pattern-level view.** APG interaction IDs already show up ad-hoc in test names (`menus-11`, `menus-13`, `checkbox-42`, `switch-1`) — the instinct exists, but there's no system behind it.
- **Contracts are copy-pasted.** Each component re-derives "a checkbox should toggle on Space, expose `aria-checked`, be labelled." That logic should be authored once per pattern and reused.
- **Gaps are invisible.** When a component partially conforms, that gap isn't recorded anywhere greppable.
- **Unused deps.** `@axe-core/playwright` and `@playwright/test` are in `package.json` but wired to nothing.

## Design

### Three-layer architecture

```
internal/aria-spec/ (private, unpublished)
├── patterns/ one contract per APG pattern, authored as DATA + assertions
│ switch.ts → expectations: [
│ { id: 'switch-role-and-checked', priority: 'blocker', run },
│ { id: 'switch-space-toggles', priority: 'blocker', run },
│ { id: 'switch-labelled', priority: 'blocker', run },
│ { id: 'switch-optionally-described', priority: 'major', run },
│ ...
│ ]
├── primitives/ keyboard / focus / aria / dom helpers (runtime-agnostic)
├── harness/ AriaHarness adapter interface (jsdom + browser implementations)
├── runner/ runContract(pattern, { harness, bindings, expectedFailures })
└── report/ emits JSON → coverage matrix (pattern × component × expectation)
```

### 1. Patterns as data, with priority tiers

Each pattern is a typed set of **expectations**, each tagged with a priority: `blocker | major | minor | optional`. CI gates on `blocker`; the rest are reported. This lets us gate meaningfully without blocking on optional-authoring niceties.

### 2. Components bind the contract + declare `expectedFailures`

The per-component test stays tiny and is **honest about gaps**:

```ts
runSwitchContract({
render: () => {}} />,
expectedFailures: [
// tracked gap, not a missing test
],
});
```

`expectedFailures` is the key adoption mechanism: we roll the contract across all 122 components immediately, and **every gap becomes an explicit, greppable, reviewable line** instead of a missing test. Burning down `expectedFailures` becomes the hardening backlog.

### 3. Coverage matrix

The runner emits `pattern × component × expectation` results → a coverage report (% of expectations passing per component, per pattern). This turns APG conformance into a dashboard metric.

## Runtime: two tiers, runtime-agnostic contracts

Expectations are authored against an abstract **`AriaHarness`** adapter (render / query-by-role / get-attribute / press-key / focus / accessible-name), so the *same* contract runs in either tier. This mirrors the proven internal ARIA-spec design (specs take an optional `device` param; default = in-process, override = real browser).

### Tier 1 — jsdom (default, fast, every PR)

Vitest + Testing Library + `user-event` — the stack our existing tests already use. Genuinely covers the majority of APG expectations: role presence, accessible-name computation, ARIA state wiring (`aria-expanded`/`-controls`/`-checked`/`-selected`/`-activedescendant`, `aria-labelledby`/`-describedby`), and keyboard behavior (arrows, Home/End, typeahead, Enter/Space, Escape, Tab moving `activeElement`). Runs in the existing `pnpm test` path.

### Tier 2 — real browser (gated, for fidelity-critical expectations)

jsdom **fakes** exactly the things composite APG widgets depend on, so a browser tier is necessary — not optional — for full conformance:

- The **real accessibility tree** as exposed to AT (jsdom only approximates it) → `toMatchAriaSnapshot`
- Focus-trap Tab **wrapping**, `inert`, native `.showModal()`, Popover API **top layer** (all currently mocked in our tests — e.g. Switch/DropdownMenu stub `showPopover`)
- `:focus-visible` and forced-colors state (CSS-engine, invisible to jsdom)

**Runner decision: Vitest Browser Mode** (`@vitest/browser` + Playwright/Chromium provider), added as a **third Vitest project** (`browser`) alongside the existing `ui`/`node` projects. Rationale: one runner, one config, same `render()`/Testing-Library API as Tier 1 — but `document` is real Chromium, and the Playwright provider gives native `toMatchAriaSnapshot` + `toHaveScreenshot`. Only fidelity-critical expectations opt into this tier.

Runner-up considered: **Storybook Test Runner** (reuses our 169 stories as fixtures, and CI already builds Storybook) — attractive for a cheap axe-per-story sweep, but it's a second test system with its own conventions. Likely a *complement* later (axe-per-story in the storybook job), not the primary contract runtime.

### Structure & visual layers

- **Structure (static):** wire **axe-core** (`vitest-axe` in Tier 1) as a per-component pass — contrast/name/dup-id/role violations. Complements, does not replace, the behavior expectations.
- **Structure (tree):** accessibility-tree snapshot (`toMatchAriaSnapshot`) — a *text* YAML of roles/names, reviewable in PRs, no pixels. Tier 2.
- **Visual (pixels):** scoped tight — screenshots **only** for inherently-visual a11y states (`:focus-visible` ring, forced-colors/high-contrast, reduced-motion). Real screenshots + browser, its **own opt-in/nightly job**, ~2 themes. Not in the blocking path.

## CI shape

Leverages the existing `check-scope` gating (jobs already skip docsite-only PRs) and the existing `build-storybook` job.

```
check-scope ──┬─► test (jsdom Tier 1, fast) ← blocker + all, every PR [existing]
├─► build-storybook ← + optional axe-per-story sweep
├─► aria-spec-browser (Tier 2) ← blocker-gated, component PRs only, cached Chromium
└─► aria-spec-visual (screenshots) ← nightly / labeled, non-blocking
```

Streamlining levers:
- **Gate Tier 2 on a `component_changed` scope output** (extend the existing changed-files step) — don't run the browser tier on non-component PRs.
- **Gate the build only on `blocker` expectations**; upload the full coverage matrix as an artifact for `major`/`minor`. Lets us enable across 122 components without a wall of red.
- **Cache the Playwright browser** (`~/.cache/ms-playwright`, keyed on Playwright version) — the install is the slow part, not the tests.
- **Chromium-only per-PR** (the a11y tree + keyboard are engine-consistent); cross-browser is a nightly concern.
- **Shard Tier 2** (`--shard=i/N`) only if/when the suite grows.

## Consolidation

Existing a11y assertions scattered across `*.test.tsx` should **move/consolidate into the pattern contracts**. The ad-hoc APG-tagged tests (`menus-*`, `checkbox-42`, `switch-1`) become the first `expectedFailures`-free contract bindings. Goal: one authoritative place per pattern, per-component tests reduced to a binding + a gap list.

## Scope of the first prototype (separate PR)

Prove the architecture end-to-end on **one pattern** before authoring all patterns:

- [ ] `internal/aria-spec/` package scaffold (`private: true`), `AriaHarness` interface + jsdom adapter
- [ ] Runner with priority tiers + `expectedFailures`, coverage-matrix emit
- [ ] Author the **Switch/Checkbox** pattern contract (existing `switch-1`/`checkbox-42` tags to migrate)
- [ ] Bind Switch (+ Checkbox) as jsdom (Tier 1) contract tests, running in the existing suite
- [ ] Wire the **Tier 2 browser project** (Vitest Browser Mode) + one browser binding proving `toMatchAriaSnapshot`
- [ ] Emit a JSON coverage matrix + human-readable summary
- [ ] Migrate the existing ad-hoc APG-tagged assertions into the new contract

## Explicitly out of scope (follow-ups)

- `vitest-axe` structural pass wiring (dep not yet installed)
- Pixel screenshots for focus/contrast states
- Authoring the remaining ~22 patterns
- Coverage dashboard UI
- Storybook axe-per-story sweep

## Open questions

- Package name / location: `@astryxdesign/aria-spec` at `internal/aria-spec/` — OK?
- Pattern authoring format — pure TS modules vs. a declarative spec object the runner interprets?
- Do we gate CI on `blocker` expectations from day one, or report-only until coverage is broad?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.