microsoft / microsoft/microsoft-ui-reactor
[Bug] AOT Selftests intermittently exits 1 with no failing test and no TAP trailer — a host crash distinct from #988, and misattributed to it
- Dominant language
- C#
- Stars
- 646
- Forks
- 54
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 84
Description
## Summary
The `AOT Selftests` leg intermittently exits 1 with **no failing test, no TAP trailer, and no watchdog message**. It reproduces on clean `main` and is distinct from both #988 (watchdog timeout) and #1037 (`*_Removed` assertion cluster).
Because #988 is the available explanation for "red AOT leg with no obvious cause", this shape gets triaged into it and disappears. **The TAP trailer is the only reliable discriminator.**
## Signature
| | `not ok` | `# Total failures` trailer | `timed out … in flight` | orphan `dotnet` at teardown |
|---|---|---|---|---|
| #988 watchdog | present | **present** | present | 6 |
| #1037 `*_Removed` | present | present | – | 6 |
| **this crash** | **0** | **0** | **0** | 6 |
### Orphaned processes are *not* a discriminator
Worth stating explicitly, because it reads like crash evidence: `Terminate orphan process: pid (…) (dotnet)` appears **6 times in every failure of every shape**, including plain assertion failures. It is a teardown artifact of any failed job, not a crash indicator. Only the trailer separates the shapes.
## Measured rate
Sampled every `AOT Selftests` job execution across recent `ci.yml` runs, `filter=all` so re-run-masked first attempts are visible:
| branch | executions | failures | rate |
|---|---|---|---|
| `main` | 101 | 4 | 3.96% |
| a feature branch (PR #1011) | 48 | 2 | 4.17% |
Of `main`'s 4, three carry a trailer (#988 / #1037 family) and **one is this crash**: `3d0a94ad`, run `29319850743` — `not ok`=0, trailer=0, timeout=0.
## Why the obvious control run cannot settle it
At the measured ~4% rate, **a single clean re-run on `main` passes 95.8% of the time even when the defect is fully present.** One control run cannot come out the other way in any useful sense, and ~70 executions are needed before a zero-failure result is 95%-confident evidence of a different rate.
This bit me directly while investigating: an initial 25-run `main` sample showed **0 failures** and pointed at "the feature branch causes it". Widening to 101 executions surfaced 4 and inverted the conclusion. At 25 draws, a 4% defect shows zero **34%** of the time — the sample size, not the code, was deciding the answer.
## Repro / triage
```bash
# every AOT Selftests execution, including re-run-masked first attempts
gh api "repos/microsoft/microsoft-ui-reactor/actions/runs//jobs?per_page=100&filter=all"
# classify: trailer present => #988/#1037 family; absent with exit 1 => this bug
gh api "repos/microsoft/microsoft-ui-reactor/actions/jobs//logs" | grep -c "# Total failures"
```
`filter=all` matters — the default returns the latest attempt only, so any failure that passed on re-run is invisible. That default makes the standard query structurally unable to observe this bug.
## Suggested next step
Have the AOT selftest host emit a trailer unconditionally (even on abnormal termination), so "no trailer" becomes an explicit, greppable *host died* signal rather than an absence that has to be inferred. Today the absence of output is doing the work of a diagnostic, which is why the shape is invisible to triage.
Found while investigating CI on PR #1011 (issue #989).
Contributor guide
Research direction
Start with the AOT selftest host and ci.yml, then use the provided GitHub Actions queries with filter=all to inspect runs and logs for missing trailers. Confirm that abnormal host termination produces an explicit trailer while preserving the distinction from watchdog and assertion failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, github-actions
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100