callstack / callstack/agent-device
ci(ios): the iOS simulator smoke lane is failing on main across several rotating signatures
- Dominant language
- TypeScript
- Stars
- 4.6k
- Forks
- 299
- Avg merge
- 10h 17m
- Merged PRs (30d)
- 515
Description
The fixture-backed iOS simulator E2E smoke job is failing frequently on `main`, with **different assertions on different runs**, which makes it unusable as a merge signal: a red smoke on a PR currently says nothing about that PR.
**Rate.** Of the last 12 `ios.yml` runs on `main` (2026-09-10 → 2026-09-11): **6 failures**, 2 success, 4 cancelled.
**Signatures observed, all on `main` or reproducing independently of a change:**
| assertion | scenario / step | seen in |
|---|---|---|
| `wait for the WebView page to expose its link` | `smoke:webview-remote-content` | main `34592935964` |
| `id="automation-longpress" did not become visible after scrolling` | scroll + `is visible` | PR run `34595714111` |
| `wait timed out for text: Agent Device Tester`, `wait_capture_stalled`, `readableCaptures: 0` on the FIRST capture after a cold `open --relaunch` | `smoke:automation-input` | main `34400074702`, `34226333904` — already tracked as #2343 |
| `xcrun --sdk iphonesimulator --show-sdk-version (spawnSync xcrun ETIMEDOUT)` after `** TEST BUILD SUCCEEDED **` | runner build step | PR run `34490275779` — matches #2422 (closed, recurring) |
| `Test timed out in 5000ms` | "Verify clean-installed Simulator snapshot bridge preparation" | PR run `34519750958` |
**The clearest evidence that this is the lane and not the change under test:** two runs of the *same commit* (`6ba5b3da81`, job `103251301238` then the re-run `103255078739`) failed with two *different* assertions — `automation-longpress` first, then the webview-link one that `main` is also failing. A deterministic regression cannot rotate its symptom.
**Why it matters now.** Reviewers are being asked to judge merge-readiness against a signal that is red for unrelated reasons, and the honest response to each red is a manual bisect against main — which is exactly the cost this lane exists to avoid. #2343 covers one signature; the others are unfiled.
**Suggested direction** (not prescriptive): step 14 runs `node --test` directly with no retry layer, while the replay steps in the same job use `--retries 2`, so a single cold-start stall fails the whole job. Worth deciding per-scenario which failures are genuinely non-retriable, and separating "the lane found a real defect" from "the simulator was cold or slow". The capture-stall class in #2343 already has a measured mechanism to build on.
Related: #2343, #2422.
Contributor guide
Assessment
This issue has not been assessed yet.