microsoft / microsoft/microsoft-ui-reactor

[Flaky] AOT Selftests: process dies mid-run at fixture 481/1459 with no TAP trailer and no `not ok`

Open
#1,152 0 comments 0 reactions 0 assignees View on GitHub
bug needs-triage
Dominant language
C#
Stars
646
Forks
54
Avg merge
1d 3h
Merged PRs (30d)
84

Description

### What happened?

The `AOT Selftests` CI leg intermittently **dies mid-run** rather than reporting a failing fixture. This is a different signature from the two AOT flakes already tracked (#1037 `ExitTr_Removed`, #990 `Issue487_SV2*`), both of which run the corpus to completion and fail on an assertion.

Observed on PR #1149 (job [98975322325](https://github.com/microsoft/microsoft-ui-reactor/actions/runs/33208400736/job/98975322325)):

| | failing run | re-run, identical commit |
|---|---|---|
| fixtures started (`# Running:`) | **481** | **1459** |
| `# Total failures:` trailer | **absent** | `# Total failures: 0` |
| `not ok` lines | **none** | none |
| exit code | 1 | 0 |

The last output is `# Fixture time: ConnectedAnimation_OrphanOnlyPassDoesNotCrash 22`; the process dies **between fixtures**, before `# Running: ThreadSafe_RapidBackgroundSetState` is printed. In the passing run that fixture is index 481, immediately after `ConnectedAnimation_OrphanOnlyPassDoesNotCrash` at 480, so the ordering is the same and the run simply stops there.

Why this matters beyond "a flake": the job fails closed here (exit 1), but the failure carries **no diagnostic at all** — no failing fixture name, no assertion text, no trailer. There are two materially different explanations and the log cannot distinguish them:

1. the host really died between fixtures, or
2. the host produced more output that was **lost**, since it ends its run with a teardown-free `TerminateProcess` (#680) and the job pipes stdout through `Tee-Object`, so anything still buffered at termination is discarded.

If it is (2), the same truncation could in principle drop a genuine `not ok` on a future run.

Not a watchdog kill: the death is **32.9s** after the first fixture (`20:34:30.749` → `20:35:03.638`), far inside the suite budget, and none of the `StartHangWatchdog` / per-fixture timeout paths emit their diagnostics.

`ThreadSafe_RapidBackgroundSetState` is not in `SelfTestRunner.DefaultAotSkipPatterns` and is a plain registry entry (`SelfTestFixtureRegistry.cs:571`, `:2278`).

### Steps to reproduce

Intermittent — it reproduced once in ~10 AOT legs on this branch and did not reproduce on re-run of the identical commit.

1. `dotnet publish tests/Reactor.AppTests.Host -p:PublishAotInternal=true -p:Platform=x64 -r win-x64 -c Release -o `
2. `& /Reactor.AppTests.Host.exe --self-test 2>&1 | Tee-Object -FilePath aot-selftest-output.tap`
3. Compare `(Select-String '# Running: ' aot-selftest-output.tap).Count` against 1459 and check for the `# Total failures:` trailer. A short count with no trailer is this bug.

A cheap mitigation regardless of root cause: have the AOT job assert the trailer is present, so a truncated stream is reported as "truncated" rather than as an unexplained exit 1.

### Reactor version / commit

f390d8f5aa71a59d137b78a5aa00987b3488de54 (PR #1149; the crash is independent of that PR — its diff only touches `tests/Reactor.PackagedTests/*.cs`, which the AOT job never compiles, since `Reactor.AppTests.Host.csproj` has no reference to `Reactor.PackagedTests`)

### Platform

x64

### .NET SDK version

10.0.x (`windows-latest` GitHub-hosted runner)

### Windows version

windows-latest GitHub-hosted runner

### Windows App SDK version

As resolved by `main` at the commit above.

### Logs / stack trace

```
# Running: ConnectedAnimation_OrphanOnlyPassDoesNotCrash
ok ConnectedAnimOrphanOnly_DestinationMounted
ok ConnectedAnimOrphanOnly_NothingStarted
ok ConnectedAnimOrphanOnly_SurvivesRepeatedRoundTrips
# Fixture time: ConnectedAnimation_OrphanOnlyPassDoesNotCrash 22
##[error]Process completed with exit code 1.
```

(Expected next line, per the passing run: `# Running: ThreadSafe_RapidBackgroundSetState`.)

### Confirmation

- [x] I have searched existing issues and this isn't a duplicate. The closest are #1037 (AOT leg, `ExitTr_Removed` — assertion failure, full run) and #990 (`Issue487_SV2*` — deterministic on non-interactive sessions, full run); neither covers a mid-run process death with no trailer.
- [x] This bug reproduces against the current `main` branch — the AOT host under test is built entirely from `main` sources at this commit.

Contributor guide

Open the contributing guide

Research direction

Start with the AOT self-test entry point and the `SelfTestRunner.DefaultAotSkipPatterns` and `SelfTestFixtureRegistry.cs` locations cited in the report; compare the fixture transition around `ThreadSafe_RapidBackgroundSetState` with the `TerminateProcess` and `Tee-Object` behavior. Reproduce using the provided `dotnet publish` and `--self-test` commands, then make the failure distinguishable and ensure the AOT job detects a missing `# Total failures:` trailer.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.