nebari-dev / nebari-dev/llm-serving-pack

Add Playwright for E2E and Accessibility Testing

Open
#152 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

priority: medium ⚡ UI/UX/DX
Dominant language
Go
Stars
3
Forks
3
Avg merge
16h 31m
Merged PRs (30d)
11

Description

Summary

Add Playwright to frontend/ for browser-level end-to-end coverage of the API key management flows, plus automated accessibility checks via @axe-core/playwright. Adds a new e2e-frontend job to .github/workflows/test.yaml so both suites gate every PR alongside the existing Vitest, Go, and Helm jobs.

Motivation
  • Current frontend coverage is Vitest + Testing Library only (src/**/*.test.tsx) — jsdom, no real browser, no real navigation, no real Keycloak bootstrap path.
  • The multi-step key lifecycle (create → one-time secret reveal in KeyCreatedDialog → revoke) spans several components and is only verified in isolation today.
  • src/auth/keycloak.ts already exposes a window.__PW_E2E_AUTH__ injection hook explicitly reserved for Playwright — the auth seam exists but nothing uses it.
  • No accessibility testing exists at all; the app ships custom dialogs, a theme toggle, and Base UI/Radix primitives whose focus trapping, labeling, and contrast are currently unverified.
  • Both light and dark themes (ThemeProvider) plus the branding/theming work merged in #145 need contrast regression protection.
  • Real-browser tests catch class of bugs jsdom cannot: focus management, scroll locking, portal rendering, CSS-dependent visibility.
Acceptance Criteria
  • @playwright/test and @axe-core/playwright added to frontend/devDependencies
  • frontend/playwright.config.ts created with a webServer block that builds/serves the app (npm run build && npm run preview) and a baseURL matching the preview port
  • Chromium project configured at minimum; trace: "on-first-retry" and retries set for CI
  • E2E specs live under frontend/e2e/ and are excluded from the Vitest include glob in vite.config.ts so npm test does not pick them up
  • Auth is stubbed via page.addInitScript setting window.__PW_E2E_AUTH__ (no live Keycloak in CI)
  • Backend /api calls stubbed with Playwright page.route fixtures — no live key-manager dependency in CI
  • E2E spec: keys list renders, including the empty state
  • E2E spec: create a key via CreateKeyDialog, assert the secret is shown once in KeyCreatedDialog and is copyable
  • E2E spec: revoke a key via KeyRowActionsRevokeKeyDialog, assert the row updates
  • E2E spec: API failure path renders ErrorBanner
  • E2E spec: theme toggle switches light/dark and persists across reload
  • A11y spec runs AxeBuilder against the main view and each open dialog, asserting zero violations for wcag2a, wcag2aa, and wcag21aa tags
  • A11y assertions run in both light and dark themes (color-contrast included, not disabled)
  • Any pre-existing violations are either fixed or recorded in a documented, time-boxed exclusion list rather than silently suppressed
  • npm run e2e and npm run e2e:ui scripts added to frontend/package.json
  • e2e-frontend job added to .github/workflows/test.yaml: checkout → actions/setup-node (using frontend/.node-version, npm cache) → npm cinpx playwright install --with-deps chromiumnpm run e2e
  • Playwright browser binaries cached in CI (keyed on the resolved Playwright version) to keep job time reasonable
  • playwright-report/, test-results/, and blob-report/ added to frontend/.gitignore
  • HTML report uploaded via actions/upload-artifact on failure, with the same pinned-SHA convention used by the other workflow steps
  • Biome config updated so frontend/e2e/ lints clean under npm run check
  • README.md (or docs/) documents how to run E2E locally, including running against a real key-manager with VITE_DEV_NO_AUTH=true
Out of Scope
  • Component-level and hook-level unit tests — already covered by Vitest + Testing Library in src/
  • Go test coverage for operator/ and key-manager/ — handled by the existing test-operator and test-key-manager jobs
  • Live Keycloak login flow testing (real IdP redirect + PKCE); this issue stubs auth at the __PW_E2E_AUTH__ seam
  • Cross-browser matrix (Firefox/WebKit) and mobile viewports — Chromium only for now, expandable later
  • Visual regression / screenshot snapshot testing
  • Manual accessibility audit (screen reader walkthroughs, keyboard-only review) — axe catches automated violations only, roughly 30–40% of WCAG issues
  • Load or performance testing of the vLLM serving path

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading frontend/package.json, vite.config.ts, src/auth/keycloak.ts, and .github/workflows/test.yaml, then run the existing frontend checks. Add the Playwright configuration, mocked E2E and AxeBuilder specs under frontend/e2e/, scripts, ignore rules, and CI job described in the acceptance criteria. Done means the Chromium E2E and light/dark accessibility suites pass locally and in every PR.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, playwright, typescript, vite
Domain
accessibility, ci-cd, documentation, frontend, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.