microsoft / microsoft/microsoft-ui-reactor

[Flaky] CompilationLoaderTests.Cold_load_under_500ms_warm_under_50ms_for_minimal_project — wall-clock budget assertion is coupled to machine load, not to the loader

Open
#1,040 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
646
Forks
54
Avg merge
1d 3h
Merged PRs (30d)
84

Description

## Summary

`Microsoft.UI.Reactor.Tests.CheckCommandTests.CompilationLoaderTests.Cold_load_under_500ms_warm_under_50ms_for_minimal_project` asserts a **wall-clock budget**. It fails whenever the machine is loaded — most reliably when the full unit suite is run immediately after a solution build — and passes on a warm re-run. The failure carries no information about the code under test.

## Observed

Full unit suite (`tests/Reactor.Tests`, x64, Debug), run right after `dotnet build Reactor.slnx`:

```
total=13140 passed=13075 FAILED=1
Failed: CheckCommandTests.CompilationLoaderTests.Cold_load_under_500ms_warm_under_50ms_for_minimal_project
"cold load took 1568.3 ms (budget 500 ms; allow 3× for CI noise on a minimal fixture)."
```

Immediately re-run alone, warm, same binaries, no rebuild:

```
--filter "FullyQualifiedName~CompilationLoaderTests"
total=9 passed=9 failed=0
Passed: Cold_load_under_500ms_warm_under_50ms_for_minimal_project
```

`1568.3 ms` is **3.1× over a budget that already carries a stated 3× CI allowance** — i.e. ~9.4× the nominal 500 ms. That magnitude is not a marginal timing wobble; it is disk/CPU contention from the build that preceded it.

## Why this is worth fixing rather than tolerating

- **It is not a race, so retries and budget bumps both miss.** Raising the budget only moves the threshold at which contention wins; it does not decouple the assertion from machine load. The failing quantity is *"how busy was this box"*, which is not a property of the compilation loader.
- **The assertion cannot come out the other way for the right reason.** A genuine regression in `CompilationLoader` and a busy disk produce the identical red. Per `AGENTS.md` § *Checks that actually prove something*, an oracle whose input is environment-derived is not reporting on the code under test.
- **It costs fleet time disproportionately.** It surfaces as `1 failed` in a 13,140-test run, which reads as a real regression until someone opens the trx. I carried this for a long stretch as *"transient unit flake, identity unknown, ~16% residual after 10 clean runs"* — and more runs would never have named it, because **repeated runs of an unnamed failure refine a rate, not an identity.** `--logger trx` named it in one run.

## Suggested directions (not prescriptive)

1. **Split the oracle.** Keep `Warm_load_returns_cached_instance_when_files_unchanged` and `Cache_invalidates_when_file_mtime_changes` — those assert *caching behaviour* and are load-independent. Drop the wall-clock arm, or
2. **Assert relative, not absolute** — e.g. `warm < cold / K`, which cancels a common load factor, or
3. **Mark it explicitly as a perf test** and exclude it from the correctness suite, so a red means what the suite implies it means.

Option 1 or 2 keeps the coverage that matters; option 3 at least stops it from contaminating the correctness signal.

## Repro

```powershell
dotnet build Reactor.slnx -c Debug -p:Platform=x64 -p:SkipSignaturesGen=true -p:CopilotSkipCliDownload=true
dotnet test tests\Reactor.Tests --no-build -c Debug -p:Platform=x64 --logger "trx;LogFileName=all.trx"
```

Run the suite immediately after the build, on a machine that has not gone idle. Build under a **short** path (a 131-char worktree path fails WinUI XAML compile with `WMC1006`/`WMC9999`/`APPX0002` for unrelated reasons).

Found while validating a `git merge origin/main` on PR #1016; unrelated to that change — the test exercises the `mur check` compilation loader and touches none of the changed files.

Contributor guide

Open the contributing guide

Research direction

Start by locating CheckCommandTests.CompilationLoaderTests.Cold_load_under_500ms_warm_under_50ms_for_minimal_project and review its neighboring cache-behavior tests. Reproduce with the supplied dotnet build and dotnet test commands, then evaluate the suggested oracle options. Done means the correctness suite no longer reports machine-load-dependent failures while caching behavior remains covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.