callstack / callstack/agent-device
iOS: dismissing a com.apple.SafariViewService-hosted sheet works via replay/manual but not via test; index and repeat each break differently
- Dominant language
- TypeScript
- Stars
- 4.6k
- Forks
- 299
- Avg merge
- 10h 17m
- Merged PRs (30d)
- 515
Description
## Summary
Driving a Maestro-compatible flow through `agent-device test` on the iOS Simulator, a step that must dismiss a system-hosted sheet — specifically iOS's "Save Password?" prompt after a form submission, served by `com.apple.SafariViewService` per agent-device's own diagnostic hints — consistently fails to reach the screen behind it, even though the identical logical action works correctly via `agent-device replay` and via manual step-by-step `press` commands in the same session. Two other approaches tried to make the dismiss-tap robust (`index` disambiguation, wrapping in a `repeat` block) each broke in their own distinct way against this same system surface.
Version: `agent-device` 0.21.1.
Environment: iOS Simulator (iPhone 17 Pro Max, iOS 26.5), macOS 25.6 (Darwin), Expo dev-client app (debug, Metro-connected), driven via the `--maestro` Maestro-compatibility engine.
This looks closely related to #2438 and its tracked follow-ups (#2450, #2451, #2485, #2489) — same underlying `com.apple.SafariViewService` cross-process surface, different concrete symptom (a plain confirmation sheet, not an `ASWebAuthenticationSession`). Posting as a separate report since the specific failure modes below (a hard `index` failure, and `repeat` silently not dispatching) don't look identical to what those describe, but they may share a root cause or fix — happy to have this folded into one of them if a maintainer sees the overlap.
## Reproduction
Flow (Maestro-compatible YAML), abbreviated to the relevant steps:
```yaml
appId: com.example.app
---
- launchApp
- tapOn: { id: "email-field" }
- inputText: someone@example.com
- tapOn: { id: "password-field" }
- inputText: some-password
- tapOn: "Sign in"
# iOS's system "Save Password?" sheet appears here, a few seconds after
# the tap, once the sign-in network round trip resolves.
- tapOn: { text: "Not Now", optional: true }
- assertVisible: "Today"
```
### What works
- The exact same steps, run via `agent-device replay --maestro --udid `, dismiss the sheet correctly and reach the screen behind it, repeatedly, across multiple runs.
- Driving the same tap manually, step by step, via `agent-device press 'text="Not Now"' --udid --settle` (or by exact `@ref`), also dismisses it correctly every time it was tried.
- `agent-device snapshot -i` while the sheet is up correctly reports it as a distinct surface:
```
A system web sign-in sheet is presented over the app, so this snapshot shows
that sheet (hosted out of the app process). Its controls are real and
interactive; complete or dismiss the sheet to return to app content.
Simulator AX snapshot inapplicable (system-surface-presented); used XCTest
to read the system surface presented over the app.
@e1 [application] "Safari"
@e2 [window] "Save Password?"
...
@e9 [button] "Not Now"
@e10 [button] "Save"
```
### What doesn't: `agent-device test`
Running the identical flow via `agent-device test .yml --maestro --udid --metro-port -e ...` consistently fails to reach the screen behind the sheet — `assertVisible: "Today"` times out with `COMMAND_FAILED`, and the final failure snapshot still shows the sign-in screen underneath (i.e. the sheet was up and never dismissed). This reproduced across roughly a dozen runs, with three different versions of the dismiss step (see below), and with `--record-video` both on and off (ruled out as a confound).
Per-step timing from one such run's `replay-timing.ndjson` (`durationMs`, `ok`):
```
tapOn "Sign in": ok, 924ms
waitForAnimationToEnd: ok, 523ms
tapOn { text: "Not Now", optional: true }: ok, 7103ms <- note the duration
assertVisible "Today": FAILED, COMMAND_FAILED, 17088ms
```
The 7-second duration on a `tapOn ... optional: true` that (per manual testing) should resolve in under a second if the sheet is present, or fail fast if it isn't, suggests something is being retried or polled internally without either succeeding or surfacing an error — the step reports `ok: true` but the sheet was not actually dismissed.
## Three specific things tried, each breaking differently
**1. `extendedWaitUntil` before the tap** — reasonable given the sheet's real, observed arrival-time variance (anywhere from under a second to several seconds after the `Sign in` tap). But it hard-fails the whole flow if the sheet doesn't appear within the timeout, which is also a real, observed outcome on this same simulator+credential pair after enough repeated runs — so this is wrong by construction for a step that must tolerate absence, not just a robustness gap.
**2. `index` to disambiguate an `AMBIGUOUS_MATCH`** — `text="Not Now"` resolves to 3 accessibility nodes for the one button (a scroll-area and a cell both inherit the button's own accessible label), which agent-device's native `press` command correctly refuses with `AMBIGUOUS_MATCH`. Adding `index: 2` (targeting what a raw snapshot dump showed as the innermost/actual `UIButton`) to the Maestro-compatible `tapOn` breaks differently and harder: it fails outright with
```
Error (REPLAY_DIVERGENCE): Replay failed at step N (tapOn "Not Now"): regular iOS
snapshot presentation requires a valid viewport
```
wrapped in `optional: true`, this error is swallowed — the tap never fires, no exception propagates, and the step reports success.
**3. Wrapping the plain, un-indexed, optional tap in `repeat`** —
```yaml
- repeat:
times: 5
commands:
- tapOn: { text: "Not Now", optional: true }
- waitForAnimationToEnd: { timeout: 1500 }
```
builds and runs without error (the `repeat` step itself reports `ok: true`), but the sheet is never actually dismissed — confirmed by the subsequent `assertVisible` failing every time this was tried, and by the fact that this is the *only* structural difference from the version that does work via `replay` (same selector, same `optional: true`, just wrapped).
## What I ruled out
- **Not the app.** The exact same tap, dismissed by hand or via `replay`, reaches the correct screen every time; the underlying sign-in and navigation are correct.
- **Not credentials or rate limiting.** A direct auth API call with the same email+password succeeded throughout, including in the same window as failing `test` runs.
- **Not `--record-video`.** Removing it from the `test` invocation did not change the outcome.
- **Not the ambiguous-match itself, in the working cases.** The plain, un-indexed `tapOn: { text: "Not Now", optional: true }` — the same selector that's ambiguous to the native `press` command — resolves and dispatches correctly under both `replay` and manual `press`. Only `index`, `repeat`, and `test` as a whole broke it.
## What would help
1. Whatever `test` does differently from `replay` for the same Maestro-compatible step against a `com.apple.SafariViewService`-hosted surface — worth a look given #2450/#2451/#2485 are already touching exactly this boundary.
2. `index` failing outright against a system-surface capture, rather than resolving or failing closed with a clearer message — right now the error text ("regular iOS snapshot presentation requires a valid viewport") gives no hint that the cause is the system-surface boundary specifically, and `optional: true` swallowing it silently makes it worse to diagnose.
3. `repeat`'s per-iteration settle/dispatch behavior against a system surface — possibly related to #2450's "distinct lineage for in-place system-surface captures" if `repeat`'s own settle comparison between iterations is what's going wrong.
Happy to share the full flow file, `replay-timing.ndjson`, and failure artifacts if useful.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Assessment
This issue has not been assessed yet.