callstack / callstack/agent-device
iOS runner smoke: alert-observation XCTests are red or flaky on main
- Dominant language
- TypeScript
- Stars
- 4.6k
- Forks
- 299
- Avg merge
- 10h 17m
- Merged PRs (30d)
- 515
Description
## Purpose
`Smoke Tests` in the `iOS` workflow is not deterministic on `main`. Two alert-observation XCTests fail on the runner host without any Apple source changing, so every PR that reaches that step has to be re-run before a reviewer can trust it.
## Observed on `main`
- `main@973b74cc14` (push run [34741577232](https://github.com/callstack/agent-device/actions/runs/34741577232), job 103682270067, step `Run targeted iOS runner XCTest regressions`), `RunnerTests+AlertObservationTests.swift:72`:
```text
XCTAssertEqual failed: ("false") is not equal to ("true")
- Optional(… code: "ALERT_DEADLINE_EXCEEDED", message: "alert dismiss exhausted its deadline")
```
Failing test: `testAlertDismissDoesNotActivateAReplacementWithTheSameTitle`.
- PR #2533 head `a16821b661` (run [34742468875](https://github.com/callstack/agent-device/actions/runs/34742468875) attempt 1, docs-only diff), same file at `:75` and `:80`:
```text
XCTAssertEqual failed: ("First actions: 0; replacement actions: 0") is not equal to ("First actions: 1; replacement actions: 0")
```
Failing test: `testAlertCannotProveAnIdenticalReplacementAndDoesNotActivateIt`. Attempt 2 of the identical run and commit passed at 06:46 UTC, so the second case is a flake rather than a regression.
Both symptoms share one shape: a presented alert reports no actions, and the assertion times out through `ALERT_DEADLINE_EXCEEDED`.
Separate symptom on the same step family, worth tracking apart from the two above: `main@8e8eeb2ced` (run [34712516058](https://github.com/callstack/agent-device/actions/runs/34712516058)) failed `Run fixture-backed iOS simulator E2E smoke` with an assertion raised at `test/integration/ios-simulator-e2e/live-automation-scenario.ts:33` (`observeFixtureHome`).
## Required behavior
- The alert-observation assertions in `apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+AlertObservationTests.swift` pass or fail on their own terms, not on simulator alert presentation timing.
- A presented system alert that the test can see must expose its button list to the probe, or the probe must report the empty-button case as a distinct typed outcome instead of consuming the whole deadline.
- Whatever the resolution, an unchanged `main` must not require a manual re-run to go green.
## Observable completion conditions
- A diagnosis recorded here naming the mechanism (alert not yet realized when the action list is read, deadline sized below host latency, or a genuine runner-side ordering bug).
- If it is timing: the probe or deadline change lands and the `iOS` workflow's `Smoke Tests` job is green across at least five consecutive `main` pushes, with no manual re-runs.
- If it is a real defect: a failing-then-passing XCTest that pins it, plus the same five-run streak.
- The `observeFixtureHome` case is either fixed or split into its own issue with the above evidence linked.
## Dependencies
- Reproduce on a hosted macOS runner or a local simulator; the failure does not appear in unit or `check:affected` runs, so this is device-facing and `docs/agents/device-verification.md` applies.
- Related history for the alert horizon: ADR-tracked alert deadline work behind `testAlertResolutionCannotBypassRequestedDeadline` and `testAlertDeadlineBeforeActivationLeavesTheOriginalUntouched`, which pass in the same runs where the two above fail.
Discovered while babysitting #2533, whose diff touched no Apple source.
Contributor guide
Assessment
This issue has not been assessed yet.