microsoft / microsoft/winappCli
[Bug]: build-and-package is flaky across branches, including main (3 undocumented flaky tests)
- Dominant language
- C#
- Stars
- 1.3k
- Forks
- 80
- Avg merge
- 3d 6h
- Merged PRs (30d)
- 51
Description
### Describe the bug
The `build-and-package` job in `build-package.yml` is failing intermittently across branches, including `main`. Each failure is a different test, and every one passes on a subsequent run. This makes a green build largely a matter of luck and forces repeated re-runs on unrelated PRs.
Observed while validating #861, which changes five files (three pipeline YAML, two PowerShell scripts), no C# and nothing invoked by the GitHub Actions build. Four consecutive runs produced four distinct failures:
**1. `EndToEndTests` — four tests fail when `winapp init` cannot restore `Microsoft.WindowsAppSDK`** ([job 105005563144](https://github.com/microsoft/winappCli/actions/runs/35152157634/job/105005563144))
```
error: Package 'Microsoft.WindowsAppSDK' is incompatible with 'all' frameworks in project '...\TestApp.csproj'.
error: - Found 0 version(s) in C:\Program Files\dotnet\library-packs
Assert.AreEqual failed. Expected:<0>. Actual:<1>. 'actual' expression: 'initExitCode'. winapp init should succeed
```
Affected: `E2E_DotNetApp_PackageShouldIncludeRuntimeDependency`, `E2E_DotNetApp_WithWin2D_PackageShouldIncludeActivatableClasses`, `E2E_DotNetApp_SelfContained_ShouldBundleRuntimeAndEmbedActivatableClassesInExe` (all `WinAppSDK Stable`), and `E2E_DotNetProject_InitDetectsCsprojAndAddsPackageReferences_ShouldSucceed` (`EndToEndTests.cs:206`, `:296`, `:369`, `:508`).
**2. `WindowsSandboxBootstrapTests.WaitForHeartbeat_NeverPublished_SurfacesTheAgentsOwnDiagnostics`** — seen on **`main`** ([job 104978506116](https://github.com/microsoft/winappCli/actions/runs/35150855703/job/104978506116))
```
Assert.ThrowsExactlyAsync failed. Expected exception type:.
Actual exception type:.
'action' expression: '() => WindowsSandboxBackend.WaitForHeartbeatAsync(_resultDirectory, Epoch, cancellation.Token)'
at WindowsSandboxBootstrapTests.cs:137
```
`TaskCanceledException` derives from `OperationCanceledException`, and which one surfaces depends on whether the cancellation is observed through a `Task` or the token directly. `ThrowsExactlyAsync` makes that an implementation detail the test cannot tolerate.
**3. `GuestCommandServerTests.Execute_DetachedProcessReturnsAfterStartAndSurvivesTheChannel`** ([job 105012269284](https://github.com/microsoft/winappCli/actions/runs/35152157634/job/105012269284))
```
Assert.IsTrue failed. 'condition' expression:
'SpinWait.SpinUntil(() => process.Disposed, TimeSpan.FromSeconds(1))'.
The agent should release the detached process after it exits.
at GuestCommandServerTests.cs:273
```
A fixed 1-second budget for a process to exit and be disposed, on a runner already running tests at `Workers: 4`.
A fourth failure, `Mp4SinkWriterEncoder_RealEncoderCoversValidationAndSuccessfulComplete` (`MF_E_SINK_NO_SAMPLES_PROCESSED`), also recurred. That one is already tracked in #834 and is not part of this report.
### To Reproduce
1. Push any branch, or look at recent `main` runs.
2. Watch `build-package.yml` -> `build-and-package`.
3. Re-run the failed job. It usually goes green, and a later run fails on a different test.
Recent `build-package.yml` history showing the pattern across branches:
```
failure 09-16T21:54 nmetulev-typed-ui-geometry-context
success 09-16T21:54 nmetulev-deterministic-explicit-ui-actions
failure 09-16T21:23 azchohfi-esrp-sign-exes-before-packaging
success 09-16T21:11 main
failure 09-16T21:10 main
success 09-16T21:09 main
failure 09-16T21:08 dependabot/nuget/src/winapp-Analyzer/...
success 09-16T19:55 azchohfi-exercise-release-connections
failure 09-16T19:54 azchohfi-exercise-release-connections
```
Note `main` failing at 21:10 between two successes, and the same branch failing then passing a minute apart.
### Expected behavior
`build-and-package` passes deterministically on a commit that has not changed the code under test, so a red build means a real regression.
Suggested directions, roughly in order of value:
- **(2)** Assert `OperationCanceledException` instead of `ThrowsExactlyAsync`, or assert on the message/diagnostics the test actually cares about. This one looks like a straightforward correctness fix rather than infrastructure tuning.
- **(3)** Replace the fixed 1-second `SpinUntil` with a generous timeout, or await an explicit signal that the process was released.
- **(1)** Decide whether these four E2E tests should depend on a live `dotnet add package` against nuget.org at all. If they must, they need retry with backoff; if not, pin against a local feed. As written, any nuget.org hiccup turns into a red build on every open PR.
There is a secondary effect worth noting: because `build-cli.ps1` aborts at the C# test step, the `scripts/tests` Pester suite downstream of it never executes. A flake in the C# tests silently skips the PowerShell test coverage entirely, so those suites are not actually being validated in CI on any run that fails this way.
### OS Version and details
GitHub-hosted `windows-latest` runner. Tests run with `Test Parallelization enabled (Workers: 4, Scope: MethodLevel)`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with build-package.yml and the named tests in EndToEndTests.cs, WindowsSandboxBootstrapTests.cs, and GuestCommandServerTests.cs. Re-run the build-and-package job and inspect the test and downstream scripts/tests behavior on a failure. Done means the job passes deterministically and the PowerShell test suite is not skipped by these C# test failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, github-actions, powershell
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100