dotnet / dotnet/macios

[CI Postmortem] Widespread LaunchTimedOut failures on iOS/tvOS simulator tests

Open
#25,299 16 comments 0 reactions 0 assignees View on GitHub
ci-postmortem copilot
Dominant language
C#
Stars
2.9k
Forks
576
Avg merge
2d 12h
Merged PRs (30d)
123

Description

## Description

Simulator-based tests (iOS and tvOS) are randomly failing with `LaunchTimedOut` across multiple test suites (linker, monotouch, fsharp, xcframework, introspection). When this occurs, typically ALL tests in the affected test suite on a given agent fail, suggesting the issue is at the agent/simulator level rather than individual test level.

macOS and Mac Catalyst tests are **never** affected (they don't use simulators).

## Mechanism

The `LaunchTimedOut` result is set when the test app fails to connect back to the TCP listener within `LaunchTimeout` (3 minutes in CI). This means either:
- The simulator fails to boot
- The app fails to install on the simulator
- The app crashes immediately at startup before establishing a TCP connection
- Network/port connectivity issues between the simulator and the host

The relevant code path: `tests/xharness/AppRunner.cs` line 287-290 sets a 3-minute timeout on `listener.ConnectedTask`, and `tests/xharness/TestReporter.cs` line 470 categorizes it as `LaunchTimedOut` when `TestExecutionStarted` is false (i.e. the listener never received a connection).

## Frequency

This has been happening frequently over the past 2+ weeks. Here are all occurrences from April 20 - May 3, 2026:

| Date | Build ID | # Failed | PR |
|------|----------|----------|----|
| 2026-04-20 | 13882756 | 23 | #25105 |
| 2026-04-20 | 13882766 | 23 | #25105 |
| 2026-04-23 | 13918176 | 4 | #25233 |
| 2026-04-24 | 13929603 | 22 | #25243 |
| 2026-04-27 | 13945394 | 6 | #25246 |
| 2026-04-27 | 13944752 | 28 | #25251 |
| 2026-04-27 | 13944778 | 24 | #25238 |
| 2026-04-27 | 13945852 | 2 | #25197 |
| 2026-04-27 | 13946259 | 2 | #25254 |
| 2026-04-28 | 13952590 | 2 | #25197 |
| 2026-04-28 | 13952530 | 21 | #25261 |
| 2026-04-28 | 13952539 | 15 | #25197 |
| 2026-04-28 | 13954991 | 2 | #25261 |
| 2026-04-28 | 13954558 | 37 | #25262 |
| 2026-04-28 | 13955255 | 6 | #25197 |
| 2026-04-28 | 13955774 | 2 | #25254 |
| 2026-04-28 | 13956330 | 4 | #25239 |
| 2026-04-28 | 13956481 | 2 | #25274 |
| 2026-04-28 | 13956372 | 24 | #25272 |
| 2026-04-28 | 13956568 | 2 | #25274 |
| 2026-04-28 | 13957432 | 11 | #25277 |
| 2026-04-28 | 13957717 | 15 | #25197 |
| 2026-04-29 | 13959788 | 15 | #25261 |
| 2026-04-29 | 13965758 | 15 | #25197 |
| 2026-04-29 | 13968350 | 2 | #25271 |
| 2026-04-29 | 13968391 | 11 | #25277 |
| 2026-04-30 | 13977914 | 16 | #25282 |
| 2026-04-30 | 13980444 | 2 | #25234 |
| 2026-04-30 | 13980447 | 2 | #25234 |
| 2026-04-30 | 13981379 | 2 | #25239 |
| 2026-04-30 | 13981376 | 2 | #25239 |
| 2026-04-30 | 13983337 | 2 | #25286 |
| 2026-04-30 | 13983369 | 2 | #25292 |
| 2026-04-30 | 13983599 | 11 | #25293 |
| 2026-04-30 | 13983087 | 16 | #25291 |
| 2026-05-01 | 13989124 | 22 | #25294 |
| 2026-05-01 | 13989221 | 4 | #25294 |
| 2026-05-02 | 13998898 | 43 | #25296 |
| 2026-05-02 | 13998894 | 22 | #25282 |
| 2026-05-03 | 14001884 | 15 | #25294 |
| 2026-05-03 | 14001881 | 6 | #25294 |

## Affected test suites

All simulator-based test suites are affected:
- **linker tests** (dont link, link sdk, link all, trimmode copy, trimmode link)
- **monotouch tests** (iOS, tvOS)
- **fsharp tests**
- **xcframework tests**
- **introspection tests**

Only iOS and tvOS simulator tests are affected — macOS and Mac Catalyst tests are never affected.

## Observations

1. When it hits, it tends to affect **all** tests in a test suite run, not just individual ones.
2. There is no correlation with any specific PR or code change — it happens across unrelated PRs.
3. The `simctl diagnose` step that could provide diagnostic info is currently gated on `system.debug == true` (line 152 of `run-tests.yml`), so diagnostic info is rarely captured.
4. The 3-minute `LaunchTimeout` in CI (`Harness.cs` line 266) may be too short if the agent is under load.

## Possible improvements

1. Always run `simctl diagnose` when a LaunchTimedOut occurs (not just when `system.debug` is true).
2. Add diagnostic logging in `TestReporter.LaunchCallback` and `AppRunner` to capture simulator state (running simulators, device availability) when the launch times out.
3. Consider increasing `LaunchTimeout` from 3 minutes to something higher (e.g., 5 minutes).
4. Log the agent name in the test result comment to help identify if specific agents are problematic.

## Originally reported

https://github.com/dotnet/macios/pull/25282#issuecomment-4363042382

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.