Infrastructure: Add Storybook end-to-end test pipelines

Summary

Turn the on-device Storybook app into a deterministic, portable test surface.
Define a story-test authoring convention, run the generated suite locally on
each desktop endpoint, and phase CI from non-interactive checks to required
macOS, Windows Fabric, and Win32 gates with truthful results and published
artifacts.

The app already exposes a story index, control channel, stable selectors, a
Windows smoke harness, and a portable driver implementation on the test-driver
branch. This task connects those pieces into the pipeline.

Goal

Give apps/storybook end-to-end test pipelines: a documented authoring convention
for story tests, deterministic locally reproducible suites per desktop endpoint, and
a phased set of CI jobs that start non-interactive and non-required and end as
required interactive desktop gates with published artifacts and an explicit flake
policy.

Stage

Stage 2 (beta) for phases 0-2; Stage 3 (production) for phases 3-4.

The Storybook app already has every ingredient except the pipeline: a story index, a
control channel, stable testID selectors, a Windows automation harness, and (on the
test-driver branch) a portable driver and a story-test manifest format. Observed:
no CI workflow in this repository mentions apps/storybook at all. This task builds
the pipeline around those ingredients and defines what "green" is allowed to mean at
each phase.

Why it matters

Observed. The agentic component library has 23 *.stories.tsx files under
packages/agentic/components/src plus Callout.stories.tsx. The only automated
on-device assertions live in a single Windows-only harness that is not run by
CI: three smoke stories
(apps/storybook/scripts/smoke-stories.json)
plus eleven focus tests added by
4777e54f
(PR #4208) as
regression coverage for a React Native Windows Fabric focus crash.

Inferred. That commit is the sharpest available argument for this task: a crash
that terminated the application on click was caught by an on-device harness that no
pipeline runs, so nothing would have prevented it from reaching main in the first
place, and nothing prevents its recurrence today.

Observed. The existing repository-wide E2E suite (apps/E2E)
targets FluentTester, not Storybook, and requires per-platform spec files, so it does
not extend to the agentic library without duplicating tests per platform.

Inferred. Beta readiness claims for the agentic components rest on Storybook
being the demonstration surface. If Storybook is not itself tested in CI, a
regression in a component, in the theme header, in the story index, or in the native
host reaches consumers undetected.

Observed current state

What exists

Capability	Evidence	State
Story index and control REST/WebSocket/MCP server	apps/storybook/storybook-server.cjs, GET /index.json, POST /select-story-sync/<storyId>, POST /send-event on 127.0.0.1:7007	Working, used by agents
Control CLI	apps/storybook/scripts/storybook-control.cjs - list, select, args, smoke	Working
Windows on-device assertions	apps/storybook/windows-tests/storybook-smoke.test.cjs via jest.windows.config.cjs, testEnvironment: '@react-native-windows/automation', WinAppDriver 1.2.1	3 smoke stories plus 11 focus tests (1 focus-movement, 10 focus-crash regression) added by 4777e54f; JSON artifacts under artifacts/windows; run manually only
Windows agent session orchestration	windows:agent, windows:agent:start, windows:agent:stop, artifacts/windows/agent-session.json	Working, records exact PIDs
Bundle-only validation (no native toolchain)	bundle:macos -> dist/index.macos.jsbundle, bundle:windows -> dist/index.windows.bundle	Working
Declared Windows CI build chain	windows:ci: yarn bundle:windows && yarn windows:generate && yarn windows:build	Observed: declared in package.json but referenced by no workflow
macOS on-device assertions	-	Observed: none; apps/storybook/scripts/ contains only Windows scripts and the Storybook control CLIs
Portable story tests	test-driver branch: parameters.desktopTest, desktop-driver stories generate, wdio.conf.ts, desktop:test:*	Not on main; see test-driver.md
Visual regression	-	Observed: none anywhere in the repository; screenshots exist only as failure evidence in apps/E2E/errorShots

CI shape available today

Observed from .github/workflows/pr.yml:

• JS PR on ubuntu-latest runs yarn lage buildci.
• macOS PR on macos-26 does yarn build, yarn bundle:macos, pod install,
  xcodebuild, then yarn e2eprep:macos and yarn e2etest:macos for
  apps/fluent-tester, and uploads apps/E2E/reports and apps/E2E/errorShots.
• Windows PR and Win32 PR on windows-latest install WinAppDriver 1.1 with
  msiexec, build/bundle, and run the corresponding E2E suites, uploading the same
  artifact paths.
• Toolchain setup uses
  microsoft/react-native-test-app/.github/actions/setup-toolchain pinned by SHA,
  with node-version: 22.

Inferred. The interactive-desktop question is therefore already answered in
practice for Windows: GitHub-hosted windows-latest runners currently execute
WinAppDriver-based UI automation for apps/E2E in this repository. Observed
caveat: the desktop driver documentation states Windows clicks use synthetic input
and fail on a locked workstation, and that a locked session still returns source and
attributes - so a Storybook job must detect and report that condition rather than
assume the runner is interactive.

Deterministic versus visual

Observed constraints that force this distinction:

• "WinAppDriver 1.2.1 can attach to this WinAppSDK window and inspect its UI
  Automation tree, but its screenshot endpoint does not reliably capture React
  Native Windows Composition content"
  (apps/storybook/README.md).
• apps/storybook/AGENTS.md: "WinAppDriver screenshots are not a reliable capture
  path for WinAppSDK Composition content... use the agent host's desktop screenshot
  tool when visual evidence is required."
• The test-driver branch excludes "visual-diff approval infrastructure" from the
  package's scope and leaves a Windows Graphics Capture fallback as an open
  decision.
• No baseline image store, approval workflow, or diffing tool exists in the
  repository.

Inferred. Screenshot comparison cannot be a pass/fail gate on Windows today.
Visual output must be treated as evidence attached to a run, not as an assertion.

Scope

Deterministic gates (pass/fail)

These are the only checks allowed to fail a build:

1. Story index integrity - the generated story index is non-empty and matches
   the expected set of story IDs across endpoints.
2. Manifest validity and digest - desktop-driver stories generate succeeds,
   and the digest recorded by each endpoint's run matches.
3. Bundle gates - bundle:macos, bundle:windows (and, when it exists,
   bundle:win32) succeed with duplicate-module enforcement intact.
4. Fake-backend contract run - the full generated suite runs against the fake
   backend with no GUI.
5. Native build gates - windows:ci (and the macOS macos:build) complete.
6. Real-endpoint story tests - every story with parameters.desktopTest passes
   its inline plan or linked spec on the endpoint under test: element existence,
   enabled/selected state, text and value, focus, and interaction outcomes read from
   the accessibility tree.
7. Readiness and lifecycle truthfulness - the run reports a terminal state, and
   an application crash or driver-host failure cannot report success.

Optional visual evidence (never a gate)

• Per-test screenshots and source.xml captured on failure into the run artifact
  directory.
• Optional full-story screenshot sweeps captured by a host-side desktop capture tool
  where available.
• Uploaded as CI artifacts for human review, explicitly labeled non-authoritative,
  and subject to the redaction and review rules below.

Phased coverage

Phase 0 - authoring and non-interactive gate (local + ubuntu-latest).
Document the story-test authoring convention next to the existing story authoring
instructions. Add story tests for a first cohort of components beyond Button. Add
a CI step that runs desktop:generate, validates the manifest, runs
desktop:test:fake, and runs bundle:macos / bundle:windows. Inferred: this
belongs in or beside the existing JS PR job because it needs no native toolchain
and no GUI.

Phase 1 - documented local real-platform runs. Make the full local loop
reproducible and documented for macOS and Windows Fabric: start the channel server,
build/launch the app, generate, run, read artifacts, stop the session. Reuse the
existing windows:agent session model for process ownership. Expand story-test
coverage to the primitives and the components used by the current smoke set.

Phase 2 - first interactive CI job, non-required. Add a Windows Fabric job on
windows-latest modeled on the existing Windows PR job: toolchain setup,
yarn build, windows:ci, WinAppDriver install, register/launch, channel server,
run the desktop suite, always-upload artifacts, and a bounded timeout. Keep it
non-required while flake rate is measured.

Phase 3 - required gates per endpoint. Promote the Windows Fabric job to
required once its flake rate meets the policy threshold; add the macOS job on
macos-26; add the Win32 job once
win32-storybook.md delivers the endpoint. All three run the
same manifest and must report the same digest.

Phase 4 - production operations. Flake policy and quarantine mechanism, artifact
retention and redaction rules, runner-capacity ownership, and a decision on the
visual-evidence pipeline and on retiring the legacy Windows Jest smoke harness.

Cross-cutting concerns to resolve in this task

• Native CI requirements. macOS needs Xcode, CocoaPods, and a WebDriverAgentMac
  build cache; Windows needs Visual Studio 2022 with RNW prerequisites plus
  WinAppDriver. Observed: the repository already installs the Windows
  prerequisites for apps/E2E and already runs pod install + xcodebuild for
  macOS, so the incremental cost is the Storybook app's own generate/build steps.
• Interactive desktop constraints. The job must fail with a distinguishable
  infrastructure error when the session is locked or non-interactive, rather than
  reporting a product failure.
• Artifacts. Publish run.json, events.ndjson, junit.xml, ownership.json,
  driver-host.log, per-test result.json/source.xml/screenshot.png, and
  native build logs from artifacts/windows/build-logs. Observed: artifact
  directories are already git-ignored (apps/storybook/.gitignore ignores
  artifacts/ and dist/).
• Flake policy. Define a retry budget per spec, a quarantine list with an expiry
  date and an owner, and a rule that quarantined coverage does not count toward exit
  criteria. Observed precedent: apps/E2E/wdio.conf.win32.js sets
  specFileRetries: 1 and bail: 1; the Storybook suite should choose its policy
  deliberately rather than inherit it.
• Security and lifecycle. Keep the run service loopback-only; never expose the
  channel server or run service to a non-loopback interface in CI; use attach mode
  for interactive sessions and launch mode only where the job owns the app; stop
  exactly the recorded PIDs. Observed: artifacts "can contain private
  application content" and must be reviewed before sharing.

Out of scope

• Building the driver itself - see test-driver.md and
  test-driver-release-readiness.md.
• Creating the Win32 endpoint - see win32-storybook.md; this
  task only adds its CI job once the endpoint exists.
• Migrating apps/E2E or the FluentTester suites.
• Image-diff baselines, approval workflows, or a hosted visual-review service; see
  Suggested Additions in the workstream README.
• Component-level unit and snapshot tests, which the Components workstream owns.

Deliverables

1. Story-test authoring documentation in apps/storybook/README.md and
   packages/agentic/components story guidance, including the deterministic-versus-
   evidence rule.
2. Story tests for a documented cohort of components and primitives beyond Button.
3. A non-interactive CI step (generate + validate + fake run + bundles) on every PR.
4. A Windows Fabric Storybook CI job, initially non-required, with artifact upload
   and a bounded timeout.
5. A macOS Storybook CI job with the same shape.
6. A written flake and quarantine policy, and an artifact retention/redaction policy.
7. A decision record on retaining or retiring the
   @react-native-windows/automation smoke harness, including the port/command
   conflict with desktop:service and how its focus-crash regression coverage is
   preserved.

Acceptance criteria

• apps/storybook documents how to add a story test, which assertions are
  deterministic gates, and that screenshots are evidence rather than assertions.
• At least eight stories across at least five components/primitives declare
  parameters.desktopTest and pass locally on one real endpoint.
• A PR-triggered CI step runs manifest generation, manifest validation,
  desktop:test:fake, bundle:macos, and bundle:windows, and fails the build
  on any of them.
• A Windows Fabric CI job builds the Storybook app with windows:ci, launches
  it, runs the real desktop suite, and uploads run.json, junit.xml,
  events.ndjson, ownership.json, and failure screenshots with
  if: always().
• The Windows job reports a distinguishable infrastructure error - not a test
  failure - when the desktop session is locked or non-interactive.
• The job stops every process it started, verified from ownership.json or the
  recorded session manifest, and leaves no orphaned Metro, channel server,
  WinAppDriver, or app process.
• A macOS CI job runs the same manifest and reports the same manifest digest as
  the Windows job.
• Flake rate for each required job is measured over a documented window and meets
  the written threshold before promotion to required.
• A quarantine mechanism exists, quarantined tests are listed with owner and
  expiry, and quarantined stories are excluded from coverage claims.
• No CI job binds the Storybook channel server or the desktop run service to a
  non-loopback address.
• yarn lage test-links passes for all modified documentation.
• The focus-crash regression coverage added by
  4777e54f
  is either still executed by a CI job or ported to desktop story tests before
  the legacy Windows harness is retired.

Dependencies and ordering

• Depends on: test-driver.md for phases 0-2;
  test-driver-release-readiness.md before any
  job becomes required; win32-storybook.md for the Win32 job
  in phase 3.
• Cross-workstream: the Components workstream's "Component Tests" task should
  define which behaviors belong in Jest component tests versus desktop story tests,
  so the same assertion is not paid for twice.
• Ordering (Inferred): non-interactive gate first (cheap, safe, always
  required), then one interactive endpoint proven end to end, then replication.
  Adding three interactive jobs simultaneously would make flake attribution
  impossible.

Risks and open decisions

Item	Evidence	Risk or decision
CI time and runner cost	Observed: pr.yml already runs six platform jobs with 60-minute timeouts; the Storybook app adds another native build per platform.	Decide whether Storybook jobs run on every PR, on a path filter, or on a schedule.
Interactive session assumptions	Observed: RNW clicks use synthetic input and fail on a locked workstation; a locked session still returns source and attributes.	Without explicit detection, a locked runner produces misleading failures.
Screenshot fidelity on Windows	Observed: WinAppDriver cannot reliably capture WinAppSDK Composition content.	Blocks any Windows visual gate; also weakens screenshot-based triage.
Two competing Windows harnesses	Observed: windows:test (Jest + @react-native-windows/automation, WinAppDriver 1.2.1) and the desktop-driver path (WinAppDriver via appium-windows-driver) "must not be run at the same time". Observed: windows:test currently carries the only regression coverage for the focus crash fixed in 4777e54f.	Decide retention or retirement before both are wired into CI; retiring it without porting that coverage would lose a proven regression guard.
WinAppDriver version skew	Observed: CI installs 1.1; the Storybook harness requires 1.2.1.	The Storybook job must install and pin its own version explicitly.
Release-build limitation on Windows	Observed: the Storybook color-picker image "is intentionally not packaged because the Yarn pnpm asset path exceeds Windows' deployment path limit".	Any offline/Release-based CI job must account for the missing asset.
Flake budget definition	Observed: no existing flake policy in the repository; apps/E2E uses specFileRetries: 1 and bail: 1.	Retries hide real intermittent product bugs if the budget is set without a quarantine and review loop.
Coverage source of truth	Observed: smoke-stories.json and the desktop manifest are two independent story allowlists.	Consolidate on the manifest, or document why both exist.

Evidence and references

Retrieved 2026-08-20.

• apps/storybook/README.md,
  apps/storybook/AGENTS.md,
  apps/storybook/package.json
• apps/storybook/jest.windows.config.cjs,
  apps/storybook/windows-tests/storybook-smoke.test.cjs,
  apps/storybook/scripts/smoke-stories.json
• apps/storybook/storybook-server.cjs,
  apps/storybook/scripts/storybook-control.cjs,
  apps/storybook/.gitignore
• .github/workflows/pr.yml - job names,
  runner labels, WinAppDriver installation, E2E steps, artifact upload.
• 4777e54f
  "Fix windows crash in components"
  (PR #4208) -
  the focus-crash fix and the Windows regression tests it added.
• apps/E2E/README.md,
  apps/E2E/package.json,
  apps/E2E/wdio.conf.win32.js,
  apps/E2E/wdio.conf.macos.js
• test-driver branch at
  8f971021:
  DESIGN.md
  (artifacts, security invariants, platform constraints),
  apps/storybook/wdio.conf.ts

Start with .github/workflows/pr.yml, apps/storybook/package.json, apps/storybook/windows-tests/storybook-smoke.test.cjs, and apps/storybook/jest.windows.config.cjs to understand the existing build and Windows test paths. Review the test-driver branch materials and issue #4258 before defining the pipeline. Done means documented story-test conventions, reproducible desktop suites, phased CI gates for the named endpoints, truthful results, artifacts, and a flake policy.