callstack / callstack/agent-device

iOS Simulator bridge and web content: follow-ups to the #2484 fence (content-class gate, probe circuit, WebContent spike)

Aperta
#2,492 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
TypeScript
Stelle
4.7k
Fork
303
Merge medio
10h 42m
PR unite (30g)
493

Descrizione

Follow-up to #2484 / #2486. That PR is a fence: the Simulator AX bridge refuses a tree that ends at a web view's `AXRemoteElement` leaf (`remote-content-boundary`) and the route serves XCTest for the rest of the app generation. This issue scopes what makes the class of defect impossible or cheaper, in the order the design consultation ranked them (two adversarial reviews plus a `codex exec` architecture pass; verdicts below are the agreed ones).

## Why we regressed

- #2279 judged the bridge on node/depth parity, ownership, and latency against native apps. The one fact that mattered — the reader snapshots one process (`apple/snapshot-bridge/README.md`) — was never a declared capability or a test.
- The failure was silent: a well-formed tree with a viewport and no truncation passes every protocol check, and the quality verdict knows sparse roots, not an on-screen web view with nothing inside.
- No gate drove web content through the default route. The WebView lab (#1397) was a manual lab. `http(s)` URLs open Safari by design, but Safari was never in the acquisition validation set.

## Tasks, ranked

### 1. Content-class gate (high value, moderate cost)

Extend the fixture-backed iOS E2E lane with a fixed screen set captured through the **default** route, asserting coarse content presence, backend-independent: native, WKWebView (exists: `smoke:webview-remote-content`), Safari on a harness-served local page (Node http server in the scenario; no network), a system sheet. Each row opens a **fresh app generation** so an earlier fallback cannot mask a broken bridge path. Add one deterministic page action with an observable result (a page ref `press`/`fill`) to the WebView row — today it checks content and warnings only (`test/integration/ios-simulator-e2e/live-webview-remote-content.ts`). Keep fallback-policy assertions (`remote-content-boundary` / `circuit-disabled`) separate from content assertions.

### 2. Launch-observation probe opens the circuit (high value, low cost)

`snapshot-observability.ts` reads the full tree (its `depth: 1` regular hint resolves to full depth in `adapter.ts`), answers `unobservable` on a web-first app, and cannot record the refusal: only `fallbackAfterFailure` in `snapshot-route.ts` opens the circuit. So a Safari `open` pays the 300 ms fixed settle **and** the first snapshot pays one more refused bridge round trip. Pass a route-owned failure-recording hook into the probe and open the circuit for `remote-content-boundary`. Keep the verdict `unobservable`; mapping it to `observable` contradicts the contract in `snapshot-observability.ts` ("the host AX bridge published its tree").

### 3. Public help text (modest value, very low cost)

`snapshot --help` prints the internal backend capability table (`src/commands/capture/snapshot.ts`, `snapshotBackendCapabilityHelp`). The #2484 reporter read `known gap deep-extension` as this bug. Drop the internal table from ordinary help; keep the user-facing depth and fallback semantics.

### 4. Spike: bounded WebContent acquisition through the bridge's remote-access client (high information value, timeboxed)

**Problem.** The bridge cannot establish page content. Before any cross-process traversal is designed, prove whether its `XCTAccessibilityFramework initForRemoteAccess` client (`SnapshotBridgeRuntime.m`) can read an associated WebContent document while keeping foreground ownership, bounded capture, and actionable node identity.

**Blocker to clear first.** `snapshotForProcess:` checks the read pid against `primaryApp` before capture, during continuations, and after (`SnapshotBridgeRuntime.m:319/326/337`). A `describe` request with the WebContent pid is rejected by **our own gate**, which proves nothing about Apple's AX. The spike must separate the foreground-owner pid from the read pid in disposable code and keep owner checks against Safari or the fixture app.

**Prove, in order.**
1. The experiment reaches the AX call (owner/read pid split; verified WebContent pid supplied manually the first time).
2. Remote access returns page semantics: a deterministic local page with unique body text, a link, and an input; require roles and content, not a non-null root or the document title. Record native error domain/code, elapsed time, automation state, bounds. Keep a successful app-pid read as the control.
3. The boundary supplies a generic target: inspect `UIAccessibilitySnapshotKeyElement` for the `AXRemoteElement` before JSON conversion; establish whether its pid denotes the host proxy or WebContent; try direct element-rooted resolution (`SnapshotBridgeCapture.m` continuation seam; note the runner re-roots retained elements via a different request API in `RunnerAXSnapshotBridge.m`). A Safari `TabDocument … WebViewProcessID=` string is discovery evidence only, never a production association contract.
4. The target identifies the correct document: two WKWebViews, a second Safari tab, and a navigation that replaces the WebContent process.
5. Composition preserves identity and geometry: acquisition stamps every node with `target.pid` (`adapter.ts`); prove node attribution, index/parent normalization, coordinate space, clipping, and a page-ref interaction through the existing runner path.
6. Recovery stays bounded: share the existing node/request/deadline budgets across all reads; keep the fence when resolution fails; cover duplicate boundaries, process disappearance, and navigation during capture.

**Non-goals.** Shipping grafting; changing default routing or the circuit; removing the fence; Safari-specific pid parsing in production; new quality vocabulary; physical devices.

**Exit criteria.**
- *No-go:* controlled reads reach AX but cannot obtain page semantics within bounds → record the exact failure, keep XCTest fallback.
- *Partial:* manually selected WebContent reads succeed but generic association or safe composition fails → report transport feasibility only.
- *Go:* both apps yield correctly associated page content, composition survives a document replacement, a page ref acts within shared bounds → open a separate implementation issue.
- *Timebox:* two engineering days; unresolved prerequisites end as inconclusive with the missing proof named.

Report: runtime/Xcode versions, native errors, call counts, elapsed time, process identities.

### 5. Truncation completeness policy (decision, then small change)

A tree cut by the bridge's 1500-node cap (`snapshot-source/limits.ts`) **above** the web view carries no remote leaf: the page is missing and only `truncated` residue discloses it. Decide when an involuntarily capped regular acquisition is publishable versus requiring fallback, for native controls beyond the cap as well as web views. Use the existing truncation residue; do not infer missing content from surviving shapes.

## Rejected

- A producer-independent "on-screen WebView with no descendants" degraded-output verdict: empty web views are legitimate, depth/filtering can remove descendants, and truncation above the web view leaves nothing to classify. At most a qualified "content unverified" observation, not degradation or fallback.
- A dedicated `SnapshotSourceFailureKind` for per-screen refusals: changes no policy today; introduce it only when a consumer needs distinct recovery behavior, and migrate the producers that already use `unsupported` for transient states together.
- Re-asking the bridge per capture instead of the per-generation circuit: charges a refused round trip to every `wait` poll on a web screen. Revisit only with measurements of native-screen cost in hybrid apps.

## Related

- `open ` on a simulator does not bind Safari to the session (`src/platform-runtime-open-target.ts` gates the fallback to physical devices); tracked separately.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia da test/integration/ios-simulator-e2e/live-webview-remote-content.ts e leggi snapshot-observability.ts, adapter.ts e snapshot-route.ts per comprendere le asserzioni esistenti per route e fallback. Per il bridge spike, esamina SnapshotBridgeRuntime.m, SnapshotBridgeCapture.m, RunnerAXSnapshotBridge.m e snapshot-source/limits.ts, usando lo scenario local-page esistente e le control reads. Il completamento richiede i test classificati e il comportamento di help, oltre a un report timeboxed con le evidenze AX specificate e l’esito dell’uscita.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
ios, node.js, objective-c, typescript
Ambito
cli, developer-experience, mobile-dev, testing
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Attiva
Chiarezza
Abbastanza chiara
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.