facebook / facebook/astryx

Vibe test: Accessibility dimension is a false positive — scores ~100 by construction, invisible to component-level a11y quality

Open
#4,145 2 comments 0 reactions 1 assignee Claimed by @ernestt View on GitHub
vibe-test
Dominant language
TypeScript
Stars
13k
Forks
1.1k
Avg merge
1d 15h
Merged PRs (30d)
690

Description

## Summary

The **Accessibility** dimension in the vibe-test universal evaluator (`internal/vibe-tests/src/universal-eval.ts`, `analyzeAccessibility()`) reports near-100 scores for Astryx targets regardless of the actual accessibility of the underlying components. The score is structurally guaranteed to be high for any component-composed output, so it does not reflect real accessibility quality and did not move at all across the recent APG/a11y hardening releases. It should be treated as the least trustworthy of the five dimensions until it's hardened or relabeled.

## What the dimension measures today

`analyzeAccessibility()` is a **static regex scan of the generated consumer code**. It starts at 100 and only subtracts for five textual anti-patterns:

| Rule | Severity | Penalty |
|---|---|---|
| `onClick` on a raw `span/div/p/td/tr/li/img/svg` without `role`/`tabIndex` | critical | −15 |
| icon-only `` with no `aria-label` / text child | critical | −15 |
| raw `` with no nearby `label`/`aria-label` | critical | −15 |
| `` with no `alt=` | moderate | −8 |
| heading level skip (e.g. h2 → h4) | minor | −3 |

There is no runtime, no rendered DOM, no axe-core, no APG assertions.

## Why this is a false positive for Astryx

Every one of those five rules can **only fire on raw HTML**. When the agent composes from Astryx primitives (``, ``, ``, etc.), the raw `` / `` / `onClick`-on-div never appears in the consumer code — it's encapsulated inside the component. So none of the penalties can trigger and the score stays pinned at ~100.

This means the dimension actually measures **"did the agent avoid raw-HTML footguns?"** (composition hygiene), not **"is the result accessible?"** For a component library that pushes a11y into the components by design, this is high **by construction** — and it was high before the hardening sprint too.

## What it structurally cannot see

Everything the recent releases fixed lives at runtime, inside the components, invisible to a text scan of consumer code:

- Focus trapping / focus restoration (Dialog, Drawer, overlays)
- Keyboard nav: arrow keys, typeahead, Escape, roving tabindex (menus, comboboxes, trees)
- `aria-controls` / `aria-expanded` / `aria-current` correctness and wiring
- Live-region announcements (`useAnnounce`), `VisuallyHidden`
- Color contrast, reduced-motion, screen-reader behavior

The APG-compliance work that consumed 50+ commits is exactly the class of thing a regex over `` can never detect. So "vibe a11y ≈ 100" and "component a11y was clearly bad" are both true — they measure different objects (composition hygiene vs. runtime component quality).

## Impact

- The a11y number is a **blind spot that flatters the exact thing it can't verify**: Astryx's own architecture guarantees a high score regardless of reality.
- It gives no signal on regressions — a component silently dropping a built-in a11y guarantee costs zero points.
- Readers reasonably interpret ~100 as "the library is accessible," which the metric does not support.

## Proposed hardenings

Roughly in order of value:

**1. Add a runtime axe-core / APG assertion pass (highest value).**
We already render targets to HTML and screenshot them (`build-previews.ts`, `screenshot-previews.ts`). Run axe-core against the rendered DOM and fold violations into the a11y dimension. This is the only option that catches focus, ARIA wiring, and contrast — the stuff the hardening sprint actually fixed. Keep it target-neutral so all four targets are scored by the same runtime checks (this should also legitimately reward Astryx over raw HTML/baseline on merit rather than by construction).

**2. Wire the `a11y-manifests/` diff into the score.**
We already generate Astryx vs. Radix/shadcn manifests (`internal/vibe-tests/a11y-manifests/`), but they don't feed any number. Track built-in-guarantee coverage per component and penalize regressions when a component drops a guarantee it previously shipped. This turns the manifests from documentation into a scored signal and makes the hardening work visible.

**3. Relabel the current static dimension (do this regardless).**
Rename the existing static-scan dimension to something like **"A11y hygiene (composition)"** so nobody reads ~100 as "the library is accessible." Cheap, honest, and prevents misreading even before 1–2 land.

**4. Stop-clamping / surface the ceiling.**
Because the score can only ever go down from 100 and Astryx output rarely trips a rule, report the number of rules that were even *eligible* to fire so a perfect score on a trivial output is distinguishable from a perfect score on a complex one.

## Acceptance criteria

- The a11y dimension is either backed by runtime checks (option 1) and/or manifest-coverage deltas (option 2), **or** clearly relabeled as composition hygiene (option 3).
- The recent APG/a11y hardening produces a measurable, non-trivial score change on at least the affected components (proof the metric now tracks reality).
- Scoring stays target-neutral and fair per the checker protocol in `internal/vibe-tests/README.md` (same logic across targets; blind to which config produced the output).

## Pointers

- `internal/vibe-tests/src/universal-eval.ts` — `analyzeAccessibility()` (the five-rule scan) and the 5-dimension scoring model
- `internal/vibe-tests/a11y-manifests/` — existing Astryx vs. baseline manifests (currently unused by scoring)
- `internal/vibe-tests/src/build-previews.ts`, `screenshot-previews.ts` — existing render pipeline an axe-core pass could hook into
- `internal/vibe-tests/README.md` — checker protocol / fairness invariants

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.