microsoft / microsoft/microsoft-ui-reactor
[Bug] Failed builds report green: documented `--no-build` loop, and `dotnet test` gates keyed on "Failed" are inverted
- Dominant language
- C#
- Stars
- 646
- Forks
- 54
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 84
Description
### What happened?
Two related ways a **failed build** produces output that a test script reads as **success**. Both affect commands documented in `AGENTS.md`, and both were hit in real sessions rather than found by inspection.
**(a) The documented `--no-build` fast loop runs the previous binary and reports success.**
`AGENTS.md:182` recommends:
```
dotnet run --project tests/Reactor.AppTests.Host --no-build -c Debug -p:Platform=x64 -- --self-test --filter ""
```
If the preceding build failed, `--no-build` silently executes the **last good binary** and prints `# Total failures: 0`. Observed independently in two different worktrees. In the second, the stale run was compared field-by-field against that same tree's genuine pass:
```
broken build: exit=1, dll not advanced
--no-build reported: ok=27 notok=0 "# Total failures: 0"
real baseline: ok=27 notok=0 "# Total failures: 0"
```
**No field in the TAP output separates them** — not the `ok` count, not a trailer.
That identity is structural, not coincidental, and it has a stated boundary: the stale binary carries the **previous tree's test inventory**. So the `ok` count is a tell **only** when the change alters the inventory (adding a fixture → count short by one). A product-only change leaves the count identical. The signal is therefore present exactly when the stale run is least dangerous, and absent exactly when it is most dangerous — the old, working code impersonating your unfixed new code. Mutation testing is entirely in the no-tell population, since mutations are product-only by construction.
**(b) `dotnet test` does not protect against this, and the obvious string gate is backwards.**
`dotnet test` builds by default, so it cannot run a stale binary. But on a build failure it emits **no test summary at all** — only raw compiler errors and exit 1. Any success criterion keyed on *absence* of failure therefore reads green. Probed on SDK 10.0.302:
| criterion | healthy run | broken build | verdict |
|---|---|---|---|
| exit code | 0 | 1 | ✅ separates |
| `Failed:\s+(\d+)` summary | present (N=0) | **absent** | ⚠️ "default 0 on no-match" → green |
| `-match 'Passed!'` | True | False | ✅ separates |
| `-match 'error'` | False | True | ✅ separates |
| **`-match 'Failed'`** | **True** | **False** | ❌ **inverted** |
The last row is the one worth internalising. The only occurrence of the word "Failed" in a **healthy** run is inside its own summary line — `Passed! - Failed: 0, Passed: 1, ...` — and .NET 10 prints **no** "Build FAILED" banner on a compile error. So a gate written as *"fail if the output mentions Failed"* **reddens the healthy run and greens the broken build**. It is not weak; it is precisely backwards.
**Why this class deserves a note rather than just a fix.** In ordinary use a false green hides a bug. In a **mutation-testing** loop it is worse: the run scores as *"mutation SURVIVED"*, which is a verdict **against your test**, and the natural remediation is to rewrite or weaken a test that was fine. Mutations that fail to compile (wrong arity, type mismatch) are common, so the population most likely to trip this is the population that then gets misdiagnosed as vacuous. The failure manufactures a false accusation against healthy code, and the remediation is destructive.
### Steps to reproduce
1. `dotnet new xunit -o proj; cd proj`
2. `dotnet test` → note `Passed! - Failed: 0, Passed: 1, ...`, exit 0.
3. Append `THIS IS NOT VALID CSHARP` to `UnitTest1.cs`.
4. `dotnet test` → raw `error CS####` lines, exit 1, **no summary line**.
5. Compare: `$out -match 'Failed'` is **True** for step 2 and **False** for step 4.
For (a): break any file in `tests/Reactor.AppTests.Host`, build (fails), then run the `AGENTS.md:182` command verbatim and observe `# Total failures: 0`.
### Suggested fix
- Add a caveat to the `AGENTS.md:182` fast-loop entry: `--no-build` after a failed build runs the previous binary and reports success. Gate on `$LASTEXITCODE` of the build, and/or assert the output assembly's `LastWriteTime` advanced.
- Add an entry under *Checks that actually prove something*: **prefer a positive success token over the absence of a failure token** — `Passed!` / `ok=N` with N > 0 / exit code, never "no `Failed` line". Note the inverted-gate example explicitly; it is counter-intuitive enough that people rediscover it.
Related note on `Copy-Item`-based mutate/restore loops: `Copy-Item` preserves the source's `LastWriteTime`, so restoring from a backup taken **before** the build gives the restored file an **older** timestamp than the build output. Verified:
```
backup LastWriteTime : 21:25:42.132
output LastWriteTime : 21:25:43.353
restored LastWriteTime : 21:25:42.132
restored source OLDER than build output? -> True
```
An incremental build after that restore is a legitimate no-op — **exit 0, nothing rebuilt** — which is the one case an exit-code check cannot see.
### Reactor version / commit
`8d3db0a271b0e5bc8043782e63c3cd8d6e6b5e89` (branch `azchohfi-fix-datagrid-shift-tab-modifier-dispatch`, 6 commits atop `main` @ `3f85ca49`)
### Platform
x64
### .NET SDK version
10.0.302
### Windows version
Windows 11 Enterprise Insider Preview (build 26310)
### Windows App SDK version
N/A — the probe in (b) is a vanilla `dotnet new xunit` project with no Windows App SDK reference. (a) reproduces on `tests/Reactor.AppTests.Host` at whatever version `main` resolves.
### Logs / stack trace
```
----- GOOD run, lines containing 'ailed' or 'assed' -----
| Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 13 ms - proj.dll (net10.0)
----- BAD run, last 6 non-empty lines -----
| Determining projects to restore...
| All projects are up-to-date for restore.
| ...\UnitTest1.cs(10,1): error CS1519: Invalid token '}' in a member declaration [...\proj.csproj]
| ...\UnitTest1.cs(11,9): error CS1002: ; expected [...\proj.csproj]
| ...\UnitTest1.cs(11,17): error CS1513: } expected [...\proj.csproj]
| ...\UnitTest1.cs(11,19): error CS0116: A namespace cannot directly contain members such as fields, methods or statements [...\proj.csproj]
BAD contains 'ailed' anywhere : False
```
### Confirmation
- [x] I have searched existing issues and this isn't a duplicate.
- [x] This bug reproduces against the current `main` branch (`AGENTS.md:182` is unchanged on `main`; the `dotnet test` probe is SDK behaviour and branch-independent).
Contributor guide
Research direction
Start with the documented fast-loop command at AGENTS.md:182 and the nearby “Checks that actually prove something” section. Reproduce the healthy and broken `dotnet test` outputs and the failed-build `--no-build` behavior described in the issue. Done means AGENTS.md warns about stale binaries and recommends positive success signals, including the inverted `Failed` gate example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, powershell
- Domain
- documentation, testing-qa
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100