microsoft / microsoft/winappCli

[Bug]: build-and-package is flaky across branches, including main (3 undocumented flaky tests)

Open
#862 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
1.3k
Forks
80
Avg merge
3d 6h
Merged PRs (30d)
51

Description

### Describe the bug

The `build-and-package` job in `build-package.yml` is failing intermittently across branches, including `main`. Each failure is a different test, and every one passes on a subsequent run. This makes a green build largely a matter of luck and forces repeated re-runs on unrelated PRs.

Observed while validating #861, which changes five files (three pipeline YAML, two PowerShell scripts), no C# and nothing invoked by the GitHub Actions build. Four consecutive runs produced four distinct failures:

**1. `EndToEndTests` — four tests fail when `winapp init` cannot restore `Microsoft.WindowsAppSDK`** ([job 105005563144](https://github.com/microsoft/winappCli/actions/runs/35152157634/job/105005563144))

```
error: Package 'Microsoft.WindowsAppSDK' is incompatible with 'all' frameworks in project '...\TestApp.csproj'.
error: - Found 0 version(s) in C:\Program Files\dotnet\library-packs
Assert.AreEqual failed. Expected:<0>. Actual:<1>. 'actual' expression: 'initExitCode'. winapp init should succeed
```

Affected: `E2E_DotNetApp_PackageShouldIncludeRuntimeDependency`, `E2E_DotNetApp_WithWin2D_PackageShouldIncludeActivatableClasses`, `E2E_DotNetApp_SelfContained_ShouldBundleRuntimeAndEmbedActivatableClassesInExe` (all `WinAppSDK Stable`), and `E2E_DotNetProject_InitDetectsCsprojAndAddsPackageReferences_ShouldSucceed` (`EndToEndTests.cs:206`, `:296`, `:369`, `:508`).

**2. `WindowsSandboxBootstrapTests.WaitForHeartbeat_NeverPublished_SurfacesTheAgentsOwnDiagnostics`** — seen on **`main`** ([job 104978506116](https://github.com/microsoft/winappCli/actions/runs/35150855703/job/104978506116))

```
Assert.ThrowsExactlyAsync failed. Expected exception type:.
Actual exception type:.
'action' expression: '() => WindowsSandboxBackend.WaitForHeartbeatAsync(_resultDirectory, Epoch, cancellation.Token)'
at WindowsSandboxBootstrapTests.cs:137
```

`TaskCanceledException` derives from `OperationCanceledException`, and which one surfaces depends on whether the cancellation is observed through a `Task` or the token directly. `ThrowsExactlyAsync` makes that an implementation detail the test cannot tolerate.

**3. `GuestCommandServerTests.Execute_DetachedProcessReturnsAfterStartAndSurvivesTheChannel`** ([job 105012269284](https://github.com/microsoft/winappCli/actions/runs/35152157634/job/105012269284))

```
Assert.IsTrue failed. 'condition' expression:
'SpinWait.SpinUntil(() => process.Disposed, TimeSpan.FromSeconds(1))'.
The agent should release the detached process after it exits.
at GuestCommandServerTests.cs:273
```

A fixed 1-second budget for a process to exit and be disposed, on a runner already running tests at `Workers: 4`.

A fourth failure, `Mp4SinkWriterEncoder_RealEncoderCoversValidationAndSuccessfulComplete` (`MF_E_SINK_NO_SAMPLES_PROCESSED`), also recurred. That one is already tracked in #834 and is not part of this report.

### To Reproduce

1. Push any branch, or look at recent `main` runs.
2. Watch `build-package.yml` -> `build-and-package`.
3. Re-run the failed job. It usually goes green, and a later run fails on a different test.

Recent `build-package.yml` history showing the pattern across branches:

```
failure 09-16T21:54 nmetulev-typed-ui-geometry-context
success 09-16T21:54 nmetulev-deterministic-explicit-ui-actions
failure 09-16T21:23 azchohfi-esrp-sign-exes-before-packaging
success 09-16T21:11 main
failure 09-16T21:10 main
success 09-16T21:09 main
failure 09-16T21:08 dependabot/nuget/src/winapp-Analyzer/...
success 09-16T19:55 azchohfi-exercise-release-connections
failure 09-16T19:54 azchohfi-exercise-release-connections
```

Note `main` failing at 21:10 between two successes, and the same branch failing then passing a minute apart.

### Expected behavior

`build-and-package` passes deterministically on a commit that has not changed the code under test, so a red build means a real regression.

Suggested directions, roughly in order of value:

- **(2)** Assert `OperationCanceledException` instead of `ThrowsExactlyAsync`, or assert on the message/diagnostics the test actually cares about. This one looks like a straightforward correctness fix rather than infrastructure tuning.
- **(3)** Replace the fixed 1-second `SpinUntil` with a generous timeout, or await an explicit signal that the process was released.
- **(1)** Decide whether these four E2E tests should depend on a live `dotnet add package` against nuget.org at all. If they must, they need retry with backoff; if not, pin against a local feed. As written, any nuget.org hiccup turns into a red build on every open PR.

There is a secondary effect worth noting: because `build-cli.ps1` aborts at the C# test step, the `scripts/tests` Pester suite downstream of it never executes. A flake in the C# tests silently skips the PowerShell test coverage entirely, so those suites are not actually being validated in CI on any run that fails this way.

### OS Version and details

GitHub-hosted `windows-latest` runner. Tests run with `Test Parallelization enabled (Workers: 4, Scope: MethodLevel)`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with build-package.yml and the named tests in EndToEndTests.cs, WindowsSandboxBootstrapTests.cs, and GuestCommandServerTests.cs. Re-run the build-and-package job and inspect the test and downstream scripts/tests behavior on a failure. Done means the job passes deterministically and the PowerShell test suite is not skipped by these C# test failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, github-actions, powershell
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.