callstack / callstack/agent-device
iOS Simulator bridge and web content: follow-ups to the #2484 fence (content-class gate, probe circuit, WebContent spike)
- Dominant language
- TypeScript
- Stars
- 4.6k
- Forks
- 299
- Avg merge
- 10h 17m
- Merged PRs (30d)
- 515
Description
Follow-up to #2484 / #2486. That PR is a fence: the Simulator AX bridge refuses a tree that ends at a web view's `AXRemoteElement` leaf (`remote-content-boundary`) and the route serves XCTest for the rest of the app generation. This issue scopes what makes the class of defect impossible or cheaper, in the order the design consultation ranked them (two adversarial reviews plus a `codex exec` architecture pass; verdicts below are the agreed ones).
## Why we regressed
- #2279 judged the bridge on node/depth parity, ownership, and latency against native apps. The one fact that mattered — the reader snapshots one process (`apple/snapshot-bridge/README.md`) — was never a declared capability or a test.
- The failure was silent: a well-formed tree with a viewport and no truncation passes every protocol check, and the quality verdict knows sparse roots, not an on-screen web view with nothing inside.
- No gate drove web content through the default route. The WebView lab (#1397) was a manual lab. `http(s)` URLs open Safari by design, but Safari was never in the acquisition validation set.
## Tasks, ranked
### 1. Content-class gate (high value, moderate cost)
Extend the fixture-backed iOS E2E lane with a fixed screen set captured through the **default** route, asserting coarse content presence, backend-independent: native, WKWebView (exists: `smoke:webview-remote-content`), Safari on a harness-served local page (Node http server in the scenario; no network), a system sheet. Each row opens a **fresh app generation** so an earlier fallback cannot mask a broken bridge path. Add one deterministic page action with an observable result (a page ref `press`/`fill`) to the WebView row — today it checks content and warnings only (`test/integration/ios-simulator-e2e/live-webview-remote-content.ts`). Keep fallback-policy assertions (`remote-content-boundary` / `circuit-disabled`) separate from content assertions.
### 2. Launch-observation probe opens the circuit (high value, low cost)
`snapshot-observability.ts` reads the full tree (its `depth: 1` regular hint resolves to full depth in `adapter.ts`), answers `unobservable` on a web-first app, and cannot record the refusal: only `fallbackAfterFailure` in `snapshot-route.ts` opens the circuit. So a Safari `open` pays the 300 ms fixed settle **and** the first snapshot pays one more refused bridge round trip. Pass a route-owned failure-recording hook into the probe and open the circuit for `remote-content-boundary`. Keep the verdict `unobservable`; mapping it to `observable` contradicts the contract in `snapshot-observability.ts` ("the host AX bridge published its tree").
### 3. Public help text (modest value, very low cost)
`snapshot --help` prints the internal backend capability table (`src/commands/capture/snapshot.ts`, `snapshotBackendCapabilityHelp`). The #2484 reporter read `known gap deep-extension` as this bug. Drop the internal table from ordinary help; keep the user-facing depth and fallback semantics.
### 4. Spike: bounded WebContent acquisition through the bridge's remote-access client (high information value, timeboxed)
**Problem.** The bridge cannot establish page content. Before any cross-process traversal is designed, prove whether its `XCTAccessibilityFramework initForRemoteAccess` client (`SnapshotBridgeRuntime.m`) can read an associated WebContent document while keeping foreground ownership, bounded capture, and actionable node identity.
**Blocker to clear first.** `snapshotForProcess:` checks the read pid against `primaryApp` before capture, during continuations, and after (`SnapshotBridgeRuntime.m:319/326/337`). A `describe` request with the WebContent pid is rejected by **our own gate**, which proves nothing about Apple's AX. The spike must separate the foreground-owner pid from the read pid in disposable code and keep owner checks against Safari or the fixture app.
**Prove, in order.**
1. The experiment reaches the AX call (owner/read pid split; verified WebContent pid supplied manually the first time).
2. Remote access returns page semantics: a deterministic local page with unique body text, a link, and an input; require roles and content, not a non-null root or the document title. Record native error domain/code, elapsed time, automation state, bounds. Keep a successful app-pid read as the control.
3. The boundary supplies a generic target: inspect `UIAccessibilitySnapshotKeyElement` for the `AXRemoteElement` before JSON conversion; establish whether its pid denotes the host proxy or WebContent; try direct element-rooted resolution (`SnapshotBridgeCapture.m` continuation seam; note the runner re-roots retained elements via a different request API in `RunnerAXSnapshotBridge.m`). A Safari `TabDocument … WebViewProcessID=` string is discovery evidence only, never a production association contract.
4. The target identifies the correct document: two WKWebViews, a second Safari tab, and a navigation that replaces the WebContent process.
5. Composition preserves identity and geometry: acquisition stamps every node with `target.pid` (`adapter.ts`); prove node attribution, index/parent normalization, coordinate space, clipping, and a page-ref interaction through the existing runner path.
6. Recovery stays bounded: share the existing node/request/deadline budgets across all reads; keep the fence when resolution fails; cover duplicate boundaries, process disappearance, and navigation during capture.
**Non-goals.** Shipping grafting; changing default routing or the circuit; removing the fence; Safari-specific pid parsing in production; new quality vocabulary; physical devices.
**Exit criteria.**
- *No-go:* controlled reads reach AX but cannot obtain page semantics within bounds → record the exact failure, keep XCTest fallback.
- *Partial:* manually selected WebContent reads succeed but generic association or safe composition fails → report transport feasibility only.
- *Go:* both apps yield correctly associated page content, composition survives a document replacement, a page ref acts within shared bounds → open a separate implementation issue.
- *Timebox:* two engineering days; unresolved prerequisites end as inconclusive with the missing proof named.
Report: runtime/Xcode versions, native errors, call counts, elapsed time, process identities.
### 5. Truncation completeness policy (decision, then small change)
A tree cut by the bridge's 1500-node cap (`snapshot-source/limits.ts`) **above** the web view carries no remote leaf: the page is missing and only `truncated` residue discloses it. Decide when an involuntarily capped regular acquisition is publishable versus requiring fallback, for native controls beyond the cap as well as web views. Use the existing truncation residue; do not infer missing content from surviving shapes.
## Rejected
- A producer-independent "on-screen WebView with no descendants" degraded-output verdict: empty web views are legitimate, depth/filtering can remove descendants, and truncation above the web view leaves nothing to classify. At most a qualified "content unverified" observation, not degradation or fallback.
- A dedicated `SnapshotSourceFailureKind` for per-screen refusals: changes no policy today; introduce it only when a consumer needs distinct recovery behavior, and migrate the producers that already use `unsupported` for transient states together.
- Re-asking the bridge per capture instead of the per-generation circuit: charges a refused round trip to every `wait` poll on a web screen. Revisit only with measurements of native-screen cost in hybrid apps.
## Related
- `open ` on a simulator does not bind Safari to the session (`src/platform-runtime-open-target.ts` gates the fallback to physical devices); tracked separately.
Contributor guide
Assessment
This issue has not been assessed yet.