microsoft / microsoft/microsoft-ui-reactor
[Flaky] AOT Selftests: process dies mid-run at fixture 481/1459 with no TAP trailer and no `not ok`
- Dominant language
- C#
- Stars
- 646
- Forks
- 54
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 84
Description
### What happened?
The `AOT Selftests` CI leg intermittently **dies mid-run** rather than reporting a failing fixture. This is a different signature from the two AOT flakes already tracked (#1037 `ExitTr_Removed`, #990 `Issue487_SV2*`), both of which run the corpus to completion and fail on an assertion.
Observed on PR #1149 (job [98975322325](https://github.com/microsoft/microsoft-ui-reactor/actions/runs/33208400736/job/98975322325)):
| | failing run | re-run, identical commit |
|---|---|---|
| fixtures started (`# Running:`) | **481** | **1459** |
| `# Total failures:` trailer | **absent** | `# Total failures: 0` |
| `not ok` lines | **none** | none |
| exit code | 1 | 0 |
The last output is `# Fixture time: ConnectedAnimation_OrphanOnlyPassDoesNotCrash 22`; the process dies **between fixtures**, before `# Running: ThreadSafe_RapidBackgroundSetState` is printed. In the passing run that fixture is index 481, immediately after `ConnectedAnimation_OrphanOnlyPassDoesNotCrash` at 480, so the ordering is the same and the run simply stops there.
Why this matters beyond "a flake": the job fails closed here (exit 1), but the failure carries **no diagnostic at all** — no failing fixture name, no assertion text, no trailer. There are two materially different explanations and the log cannot distinguish them:
1. the host really died between fixtures, or
2. the host produced more output that was **lost**, since it ends its run with a teardown-free `TerminateProcess` (#680) and the job pipes stdout through `Tee-Object`, so anything still buffered at termination is discarded.
If it is (2), the same truncation could in principle drop a genuine `not ok` on a future run.
Not a watchdog kill: the death is **32.9s** after the first fixture (`20:34:30.749` → `20:35:03.638`), far inside the suite budget, and none of the `StartHangWatchdog` / per-fixture timeout paths emit their diagnostics.
`ThreadSafe_RapidBackgroundSetState` is not in `SelfTestRunner.DefaultAotSkipPatterns` and is a plain registry entry (`SelfTestFixtureRegistry.cs:571`, `:2278`).
### Steps to reproduce
Intermittent — it reproduced once in ~10 AOT legs on this branch and did not reproduce on re-run of the identical commit.
1. `dotnet publish tests/Reactor.AppTests.Host -p:PublishAotInternal=true -p:Platform=x64 -r win-x64 -c Release -o `
2. `& /Reactor.AppTests.Host.exe --self-test 2>&1 | Tee-Object -FilePath aot-selftest-output.tap`
3. Compare `(Select-String '# Running: ' aot-selftest-output.tap).Count` against 1459 and check for the `# Total failures:` trailer. A short count with no trailer is this bug.
A cheap mitigation regardless of root cause: have the AOT job assert the trailer is present, so a truncated stream is reported as "truncated" rather than as an unexplained exit 1.
### Reactor version / commit
f390d8f5aa71a59d137b78a5aa00987b3488de54 (PR #1149; the crash is independent of that PR — its diff only touches `tests/Reactor.PackagedTests/*.cs`, which the AOT job never compiles, since `Reactor.AppTests.Host.csproj` has no reference to `Reactor.PackagedTests`)
### Platform
x64
### .NET SDK version
10.0.x (`windows-latest` GitHub-hosted runner)
### Windows version
windows-latest GitHub-hosted runner
### Windows App SDK version
As resolved by `main` at the commit above.
### Logs / stack trace
```
# Running: ConnectedAnimation_OrphanOnlyPassDoesNotCrash
ok ConnectedAnimOrphanOnly_DestinationMounted
ok ConnectedAnimOrphanOnly_NothingStarted
ok ConnectedAnimOrphanOnly_SurvivesRepeatedRoundTrips
# Fixture time: ConnectedAnimation_OrphanOnlyPassDoesNotCrash 22
##[error]Process completed with exit code 1.
```
(Expected next line, per the passing run: `# Running: ThreadSafe_RapidBackgroundSetState`.)
### Confirmation
- [x] I have searched existing issues and this isn't a duplicate. The closest are #1037 (AOT leg, `ExitTr_Removed` — assertion failure, full run) and #990 (`Issue487_SV2*` — deterministic on non-interactive sessions, full run); neither covers a mid-run process death with no trailer.
- [x] This bug reproduces against the current `main` branch — the AOT host under test is built entirely from `main` sources at this commit.
Contributor guide
Research direction
Start with the AOT self-test entry point and the `SelfTestRunner.DefaultAotSkipPatterns` and `SelfTestFixtureRegistry.cs` locations cited in the report; compare the fixture transition around `ThreadSafe_RapidBackgroundSetState` with the `TerminateProcess` and `Tee-Object` behavior. Reproduce using the provided `dotnet publish` and `--self-test` commands, then make the failure distinguishable and ensure the AOT job detects a missing `# Total failures:` trailer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100