callstack / callstack/agent-device

iOS runner smoke: alert-observation XCTests are red or flaky on main

Open
#2,546 0 comments 0 reactions 0 assignees View on GitHub
bug needs-triage
Dominant language
TypeScript
Stars
4.6k
Forks
299
Avg merge
10h 17m
Merged PRs (30d)
515

Description

## Purpose

`Smoke Tests` in the `iOS` workflow is not deterministic on `main`. Two alert-observation XCTests fail on the runner host without any Apple source changing, so every PR that reaches that step has to be re-run before a reviewer can trust it.

## Observed on `main`

- `main@973b74cc14` (push run [34741577232](https://github.com/callstack/agent-device/actions/runs/34741577232), job 103682270067, step `Run targeted iOS runner XCTest regressions`), `RunnerTests+AlertObservationTests.swift:72`:

```text
XCTAssertEqual failed: ("false") is not equal to ("true")
- Optional(… code: "ALERT_DEADLINE_EXCEEDED", message: "alert dismiss exhausted its deadline")
```

Failing test: `testAlertDismissDoesNotActivateAReplacementWithTheSameTitle`.

- PR #2533 head `a16821b661` (run [34742468875](https://github.com/callstack/agent-device/actions/runs/34742468875) attempt 1, docs-only diff), same file at `:75` and `:80`:

```text
XCTAssertEqual failed: ("First actions: 0; replacement actions: 0") is not equal to ("First actions: 1; replacement actions: 0")
```

Failing test: `testAlertCannotProveAnIdenticalReplacementAndDoesNotActivateIt`. Attempt 2 of the identical run and commit passed at 06:46 UTC, so the second case is a flake rather than a regression.

Both symptoms share one shape: a presented alert reports no actions, and the assertion times out through `ALERT_DEADLINE_EXCEEDED`.

Separate symptom on the same step family, worth tracking apart from the two above: `main@8e8eeb2ced` (run [34712516058](https://github.com/callstack/agent-device/actions/runs/34712516058)) failed `Run fixture-backed iOS simulator E2E smoke` with an assertion raised at `test/integration/ios-simulator-e2e/live-automation-scenario.ts:33` (`observeFixtureHome`).

## Required behavior

- The alert-observation assertions in `apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+AlertObservationTests.swift` pass or fail on their own terms, not on simulator alert presentation timing.
- A presented system alert that the test can see must expose its button list to the probe, or the probe must report the empty-button case as a distinct typed outcome instead of consuming the whole deadline.
- Whatever the resolution, an unchanged `main` must not require a manual re-run to go green.

## Observable completion conditions

- A diagnosis recorded here naming the mechanism (alert not yet realized when the action list is read, deadline sized below host latency, or a genuine runner-side ordering bug).
- If it is timing: the probe or deadline change lands and the `iOS` workflow's `Smoke Tests` job is green across at least five consecutive `main` pushes, with no manual re-runs.
- If it is a real defect: a failing-then-passing XCTest that pins it, plus the same five-run streak.
- The `observeFixtureHome` case is either fixed or split into its own issue with the above evidence linked.

## Dependencies

- Reproduce on a hosted macOS runner or a local simulator; the failure does not appear in unit or `check:affected` runs, so this is device-facing and `docs/agents/device-verification.md` applies.
- Related history for the alert horizon: ADR-tracked alert deadline work behind `testAlertResolutionCannotBypassRequestedDeadline` and `testAlertDeadlineBeforeActivationLeavesTheOriginalUntouched`, which pass in the same runs where the two above fail.

Discovered while babysitting #2533, whose diff touched no Apple source.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.