microsoft / microsoft/microsoft-ui-reactor

Flaky: `ExitTr_Removed` + 3 sibling exit-transition fixtures fail intermittently in the AOT Selftests leg (reproduces on `main`)

Open
#1,037 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
646
Forks
54
Avg merge
1d 3h
Merged PRs (30d)
84

Description

Found while establishing attribution for an AOT Selftests failure on #1011. **It is not branch-local — it reproduces on `main`.** Filing so the next person who hits it doesn't spend the hour I nearly did.

## The signature

```
not ok ExitTr_Removed - assertion failed
not ok ExitTr_RemovedAgain - assertion failed
not ok CondExit_Removed - assertion failed
not ok Asym_Removed - assertion failed
```

All four live in `tests/Reactor.AppTests.Host/SelfTest/Fixtures/ReconcilerBigCoverageFixtures.cs` and share one shape: a budget-bounded `Harness.WaitFor(() => H.FindText(...) is null, ...)` immediately after triggering a **Fade exit transition**.

## Sightings

Swept every CI run since 2026-07-28, checking **all attempts** per run:

| run | branch | head | date | failed |
|---|---|---|---|---|
| `30408541779` | **`main`** | `d08fc786` | 07-28 23:36 | `ExitTr_Removed` |
| `30585946369` | `azchohfi-analyzer-noop-modifier` | `866173f0` | 07-30 22:06 | `ExitTr_Removed` |
| `30671252520` | `azchohfi-guard-doc-screenshot-overwrites` | `f8953661` | 07-31 23:02 | all four |

Three sightings in 511 runs with an AOT Selftests job — roughly **0.6%**. All were green on re-run with no push in between, so it is intermittent, not deterministic.

**Sweeping for these runs by `conclusion == failure` will miss them.** `30671252520`'s overall conclusion is now `success`, because a re-run replaced it; the failure only exists in attempt 1. Any ledger built from failed runs has a hole exactly the size of "someone hit re-run".

## It is AOT-specific

Same regex, same window, the other leg:

| leg | job attempts | failures | sightings of this signature |
|---|---|---|---|
| `AOT Selftests` | 511 | 25 | **3** |
| `Selftests` (non-AOT) | 541 | **23** | **0** |

The failure counts are near-identical, so the non-AOT leg was not simply quiet — 23 failures existed and were searched with a regex proven to match (it matched in the AOT leg). This rules out "those fixtures are generally flaky" and isolates it to the AOT leg.

## Mechanism — a hypothesis, with the observation that constrains it

The fixture comment already names the suspect:

```csharp
// poll with per-pass wall-clock time so we converge as soon as the transition
// completes rather than betting on one fixed wait (which flakes under
// NativeAOT cold-start jitter — see issue #715).
```

Budgets against a ~300 ms transition:

| check | budget | order in run |
|---|---|---|
| `ExitTr_Removed` | 12 × 100 ms = **1200 ms** | first (23:02:03) |
| `ExitTr_RemovedAgain` | 1200 ms | 23:02:04 |
| `CondExit_Removed` | 1200 ms | 23:02:06 |
| `Asym_Removed` | 20 × 30 ms = **600 ms** | 23:02:07 |

**The interesting part: the check with the largest budget is the one that fails alone, and the check with the smallest budget only fails when all four do.** A pure per-check budget shortfall predicts the opposite — `Asym_Removed`, at half the margin, should be the first to go. What the data fits instead is a **one-time cost paid at the first exit transition after AOT cold start** (composition/animation warm-up, first-touch paging), which `ExitTr_Removed` absorbs by running first. Mild jitter → only it misses; severe → the whole cluster misses.

That is a hypothesis, not a measurement. I have not instrumented it.

## Please don't just raise the budgets

Per `AGENTS.md` § *Checks that actually prove something*: raising a budget and still failing refutes "we sampled too early" and nothing else. It does not touch a completion callback that never fires, or an ordering race decided before the first poll — both budget-insensitive, and both consistent with the evidence above.

**The check that discriminates is to log the input, not the verdict.** At poll exhaustion, record the elapsed time and whether the element is still present, then keep watching:

- element gone at ~1300 ms → genuine budget shortfall; a bigger budget is the right fix.
- element still present at 10 s → the transition never completed, and no budget ever fixes it.

Those two have identical failure text today, which is why three sightings have produced no diagnosis.

## Not to be confused with

- **The `CenterOnCurrent_UsesCursorMonitor` / `PersistPlacement_FallbackWhenEmpty` pair** — those fail together, are cursor/monitor-dependent, and are deterministic in non-interactive sessions.
- **The host-crash shape** (#978) — emits no TAP trailer and picks an arbitrary victim. This suite ran to completion; everything after the four failures passed.
- **The watchdog-kill shape** (#988) — truncates the run and marks everything downstream Skipped. Not what happened here.

Contributor guide

Open the contributing guide

Research direction

Start with tests/Reactor.AppTests.Host/SelfTest/Fixtures/ReconcilerBigCoverageFixtures.cs and the AOT Selftests leg, then read AGENTS.md § “Checks that actually prove something.” Instrument the poll exhaustion path to record elapsed time and element presence while continuing to watch. Done means distinguishing a late transition from one that never completes and documenting the observed cause before changing budgets or synchronization.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.