callstack / callstack/agent-device

iOS runner smoke: alert-observation XCTests are red or flaky on main

Aperta
#2,546 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug needs-triage
Lingua principale
TypeScript
Stelle
4.6k
Fork
299
Merge medio
10h 42m
PR unite (30g)
493

Descrizione

## Purpose

`Smoke Tests` in the `iOS` workflow is not deterministic on `main`. Two alert-observation XCTests fail on the runner host without any Apple source changing, so every PR that reaches that step has to be re-run before a reviewer can trust it.

## Observed on `main`

- `main@973b74cc14` (push run [34741577232](https://github.com/callstack/agent-device/actions/runs/34741577232), job 103682270067, step `Run targeted iOS runner XCTest regressions`), `RunnerTests+AlertObservationTests.swift:72`:

```text
XCTAssertEqual failed: ("false") is not equal to ("true")
- Optional(… code: "ALERT_DEADLINE_EXCEEDED", message: "alert dismiss exhausted its deadline")
```

Failing test: `testAlertDismissDoesNotActivateAReplacementWithTheSameTitle`.

- PR #2533 head `a16821b661` (run [34742468875](https://github.com/callstack/agent-device/actions/runs/34742468875) attempt 1, docs-only diff), same file at `:75` and `:80`:

```text
XCTAssertEqual failed: ("First actions: 0; replacement actions: 0") is not equal to ("First actions: 1; replacement actions: 0")
```

Failing test: `testAlertCannotProveAnIdenticalReplacementAndDoesNotActivateIt`. Attempt 2 of the identical run and commit passed at 06:46 UTC, so the second case is a flake rather than a regression.

Both symptoms share one shape: a presented alert reports no actions, and the assertion times out through `ALERT_DEADLINE_EXCEEDED`.

Separate symptom on the same step family, worth tracking apart from the two above: `main@8e8eeb2ced` (run [34712516058](https://github.com/callstack/agent-device/actions/runs/34712516058)) failed `Run fixture-backed iOS simulator E2E smoke` with an assertion raised at `test/integration/ios-simulator-e2e/live-automation-scenario.ts:33` (`observeFixtureHome`).

## Required behavior

- The alert-observation assertions in `apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+AlertObservationTests.swift` pass or fail on their own terms, not on simulator alert presentation timing.
- A presented system alert that the test can see must expose its button list to the probe, or the probe must report the empty-button case as a distinct typed outcome instead of consuming the whole deadline.
- Whatever the resolution, an unchanged `main` must not require a manual re-run to go green.

## Observable completion conditions

- A diagnosis recorded here naming the mechanism (alert not yet realized when the action list is read, deadline sized below host latency, or a genuine runner-side ordering bug).
- If it is timing: the probe or deadline change lands and the `iOS` workflow's `Smoke Tests` job is green across at least five consecutive `main` pushes, with no manual re-runs.
- If it is a real defect: a failing-then-passing XCTest that pins it, plus the same five-run streak.
- The `observeFixtureHome` case is either fixed or split into its own issue with the above evidence linked.

## Dependencies

- Reproduce on a hosted macOS runner or a local simulator; the failure does not appear in unit or `check:affected` runs, so this is device-facing and `docs/agents/device-verification.md` applies.
- Related history for the alert horizon: ADR-tracked alert deadline work behind `testAlertResolutionCannotBypassRequestedDeadline` and `testAlertDeadlineBeforeActivationLeavesTheOriginalUntouched`, which pass in the same runs where the two above fail.

Discovered while babysitting #2533, whose diff touched no Apple source.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia da apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+AlertObservationTests.swift ed esegui i due XCTests di osservazione degli alert indicati su un runner macOS ospitato. Leggi i test correlati alle scadenze degli alert e docs/agents/device-verification.md, quindi determina se la temporizzazione del probe, la dimensione della deadline o l'ordine del runner spiega l'elenco di azioni vuoto. Il lavoro è completato quando è disponibile un fix o una diagnosi definita, sono state completate cinque esecuzioni consecutive riuscite di iOS Smoke Tests senza riesecuzioni e observeFixtureHome è stato corretto o suddiviso in un issue separato.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
ios, swift
Ambito
mobile-dev, testing
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Attiva
Chiarezza
Abbastanza chiara
Idoneità per principianti
48/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.