microsoft / microsoft/microsoft-ui-reactor
[Flaky] CompilationLoaderTests.Cold_load_under_500ms_warm_under_50ms_for_minimal_project — wall-clock budget assertion is coupled to machine load, not to the loader
- Dominant language
- C#
- Stars
- 646
- Forks
- 54
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 84
Description
## Summary
`Microsoft.UI.Reactor.Tests.CheckCommandTests.CompilationLoaderTests.Cold_load_under_500ms_warm_under_50ms_for_minimal_project` asserts a **wall-clock budget**. It fails whenever the machine is loaded — most reliably when the full unit suite is run immediately after a solution build — and passes on a warm re-run. The failure carries no information about the code under test.
## Observed
Full unit suite (`tests/Reactor.Tests`, x64, Debug), run right after `dotnet build Reactor.slnx`:
```
total=13140 passed=13075 FAILED=1
Failed: CheckCommandTests.CompilationLoaderTests.Cold_load_under_500ms_warm_under_50ms_for_minimal_project
"cold load took 1568.3 ms (budget 500 ms; allow 3× for CI noise on a minimal fixture)."
```
Immediately re-run alone, warm, same binaries, no rebuild:
```
--filter "FullyQualifiedName~CompilationLoaderTests"
total=9 passed=9 failed=0
Passed: Cold_load_under_500ms_warm_under_50ms_for_minimal_project
```
`1568.3 ms` is **3.1× over a budget that already carries a stated 3× CI allowance** — i.e. ~9.4× the nominal 500 ms. That magnitude is not a marginal timing wobble; it is disk/CPU contention from the build that preceded it.
## Why this is worth fixing rather than tolerating
- **It is not a race, so retries and budget bumps both miss.** Raising the budget only moves the threshold at which contention wins; it does not decouple the assertion from machine load. The failing quantity is *"how busy was this box"*, which is not a property of the compilation loader.
- **The assertion cannot come out the other way for the right reason.** A genuine regression in `CompilationLoader` and a busy disk produce the identical red. Per `AGENTS.md` § *Checks that actually prove something*, an oracle whose input is environment-derived is not reporting on the code under test.
- **It costs fleet time disproportionately.** It surfaces as `1 failed` in a 13,140-test run, which reads as a real regression until someone opens the trx. I carried this for a long stretch as *"transient unit flake, identity unknown, ~16% residual after 10 clean runs"* — and more runs would never have named it, because **repeated runs of an unnamed failure refine a rate, not an identity.** `--logger trx` named it in one run.
## Suggested directions (not prescriptive)
1. **Split the oracle.** Keep `Warm_load_returns_cached_instance_when_files_unchanged` and `Cache_invalidates_when_file_mtime_changes` — those assert *caching behaviour* and are load-independent. Drop the wall-clock arm, or
2. **Assert relative, not absolute** — e.g. `warm < cold / K`, which cancels a common load factor, or
3. **Mark it explicitly as a perf test** and exclude it from the correctness suite, so a red means what the suite implies it means.
Option 1 or 2 keeps the coverage that matters; option 3 at least stops it from contaminating the correctness signal.
## Repro
```powershell
dotnet build Reactor.slnx -c Debug -p:Platform=x64 -p:SkipSignaturesGen=true -p:CopilotSkipCliDownload=true
dotnet test tests\Reactor.Tests --no-build -c Debug -p:Platform=x64 --logger "trx;LogFileName=all.trx"
```
Run the suite immediately after the build, on a machine that has not gone idle. Build under a **short** path (a 131-char worktree path fails WinUI XAML compile with `WMC1006`/`WMC9999`/`APPX0002` for unrelated reasons).
Found while validating a `git merge origin/main` on PR #1016; unrelated to that change — the test exercises the `mur check` compilation loader and touches none of the changed files.
Contributor guide
Research direction
Start by locating CheckCommandTests.CompilationLoaderTests.Cold_load_under_500ms_warm_under_50ms_for_minimal_project and review its neighboring cache-behavior tests. Reproduce with the supplied dotnet build and dotnet test commands, then evaluate the suggested oracle options. Done means the correctness suite no longer reports machine-load-dependent failures while caching behavior remains covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100