📈 CI Daily Pulse

# 🔴 main red at tip · 🔴 release/14.0 red at tip · 🔴 outerloop red at tip · 🔴 internal main red at tip · 🔴 PR completed tip red (47%) — **6 items for a dev**

> ### 🔴 For a dev — FluentUI package validation is repeatedly breaking rolling `main`
>
> **What:** The current FluentUI package fails NuGet signature validation during Windows restores. The failure is **currently deterministic across four consecutive rolling pushes**.
>
> **Do:** A dependency/feed dev should fix the package timestamp/signature or publish a consumable version to the mapped feed. [failed run](https://github.com/microsoft/aspire/actions/runs/34519184641) · [`Directory.Packages.props:10`](https://github.com/microsoft/aspire/blob/9c2af96f9163e2d89c2e9e789f5d40c395606f14/Directory.Packages.props#L10)
> - 🔴 Homebrew cask validation intermittently fails because the generated cask uses a deprecated parameter; it failed 2 of the last 4 pushes and is red at the tip ([run](https://github.com/microsoft/aspire/actions/runs/34519184641)).
> - 🔴 Outerloop dashboard checks still fail broadly on both OSes after all four attempts ([run](https://github.com/microsoft/aspire/actions/runs/34428450548)).
> - 🔴 Maven still cannot compile the staged Java sibling on Linux or Windows after all four attempts ([run](https://github.com/microsoft/aspire/actions/runs/34428450548)).
> - 🔴 Agent initialization still rejects the Playwright CLI package provenance after all four attempts ([run](https://github.com/microsoft/aspire/actions/runs/34428450548)).
> - 🔴 The Redis starter sample still never exposes its expected dashboard resources after all four attempts ([run](https://github.com/microsoft/aspire/actions/runs/34428450548)).
>
> _{The first `release/14.0` run still has two single-run failures, internal `main` has reached ten consecutive failures with its cause unconfirmed, and the latest completed PR run is red while a fresh bot rerun is active — see § 🧩 below.}

_{Window: Wed Sep 9 09:55 → Thu Sep 10 21:55 UTC · deltas vs 7-day average · Sep 10, 2026.}

### 📊 Last 36h by lane — activity, pass rate & Δ vs 7d

| Lane | 36h activity | 36h pass rate (Δ vs 7d) | Latest build |
| --- | --- | --- | --- |
| [**🌳 GH CI — main**](https://github.com/microsoft/aspire/actions/workflows/ci.yml?query=event%3Apush+branch%3Amain) | 🟩🟥🟥🟥🟥🟥🟥 | 🔴 8% (1/12) · ▼ 4.2 | 🔴 [Sep 10 19:14](https://github.com/microsoft/aspire/actions/runs/34519184641) |
| [**🌿 GH CI — release/14.0**](https://github.com/microsoft/aspire/actions/workflows/ci.yml?query=event%3Apush+branch%3Arelease%2F14.0) | 🟥🟥 | 🔴 0% (0/1, 95% CI 0–79%) · ▬ 0.0 | 🔴 [Sep 10 17:25](https://github.com/microsoft/aspire/actions/runs/34508166356) |
| [**🧪 GH CI outerloop — main**](https://github.com/microsoft/aspire/actions/workflows/tests-outerloop.yml?query=event%3Aschedule+branch%3Amain) | 🟥🟥 | 🔴 0% (0/1, 95% CI 0–79%) · ▬ 0.0 | 🔴 [Sep 10 02:11](https://github.com/microsoft/aspire/actions/runs/34428450548) |
| [**🏗️ Internal (AzDO) — main**](https://dev.azure.com/dnceng/internal/_build?definitionId=1602) | 🟥🟥🟥🟥🟥🟥 | 🔴 0% (0/9, 95% CI 0–30%) · ▼ 43.5 | 🔴 [Sep 10 19:33](https://dev.azure.com/dnceng/internal/_build/results?buildId=3072363&view=results) |
| [**🔀 GH CI — PR validation**](https://github.com/microsoft/aspire/actions/workflows/ci.yml?query=event%3Apull_request) ⭐ | 🟩🟩🟩🟩🟩🟥🟥🟥🟥🟥🟥 | 🔴 47% (34/73) · ▼ 17.3 | 🔴 [Sep 10 20:31](https://github.com/microsoft/aspire/actions/runs/34526846888) |

🧱 infra share: unavailable this run. Internal `main` has ten consecutive failed builds and remains red at the tip at [build 3072363](https://dev.azure.com/dnceng/internal/_build/results?buildId=3072363&view=results).

_{bar length = sample size / confidence (short = read its % with care) · `Δ vs 7d` = 36h rate minus the lane's 7-day average · Latest build = red/green of the lane's most recent completed in-window run, linked to it · lane names link to the workflow / pipeline.}

### 🧩 What's broken & why

**Six things need a human: rolling `main` has two repeatable package/installer breaks, and four independent outerloop families exhausted every retry.**

| | What's broken | Who's on it | Since |
| --- | --- | --- | --- |
| 🔴 | The current FluentUI package fails restore validation in four consecutive rolling `main` [runs](https://github.com/microsoft/aspire/actions/runs/34519184641). | Needs a dependency/feed dev — no auto-rerun on rolling | repeated in the last 4 pushes; red at tip |
| 🔴 | The generated Homebrew cask intermittently fails audit because it emits a parameter Homebrew has deprecated ([run](https://github.com/microsoft/aspire/actions/runs/34519184641)). | Needs an installer dev — no auto-rerun on rolling | failed 2 of the last 4 pushes; red at tip |
| 🔴 | Dashboard UI checks remain broadly broken on both OSes in the fourth-attempt outerloop [jobs](https://github.com/microsoft/aspire/actions/runs/34428450548). | Needs a dashboard dev — all outerloop retries exhausted | last failed ~20h ago; tip still red |
| 🔴 | Maven cannot compile the staged Java sibling on Linux or Windows in the outerloop [run](https://github.com/microsoft/aspire/actions/runs/34428450548). | Needs a CLI/Java dev — failed through all four attempts | last failed ~20h ago; tip still red |
| 🔴 | Agent initialization rejects the upstream Playwright CLI package across the outerloop [checks](https://github.com/microsoft/aspire/actions/runs/34428450548). | Needs a CLI/package dev — tracked in [#19159](https://github.com/microsoft/aspire/issues/19159); retries exhausted | last failed ~20h ago; tip still red |
| 🔴 | The Redis starter sample starts, but its expected resources never appear in the dashboard in the outerloop [run](https://github.com/microsoft/aspire/actions/runs/34428450548). | Needs a templates/runtime dev — retries exhausted | last failed ~20h ago; tip still red |
| 🟡 | The first `release/14.0` run passed all Kafka tests but its Windows test process exited abnormally afterward ([job](https://github.com/microsoft/aspire/actions/runs/34508166356/job/102976255265)). | Watch — too early to call; no auto-rerun on rolling release | first seen ~4.5h ago; red at tip |
| 🟡 | The same `release/14.0` run timed out waiting for a Deno debugger breakpoint ([job](https://github.com/microsoft/aspire/actions/runs/34508166356/job/102980021974)). | Watch — too early to call; no auto-rerun on rolling release | first seen ~4.5h ago; red at tip |
| 🟡 | Internal `main` is red at the tip after ten consecutive failed [builds](https://dev.azure.com/dnceng/internal/_build/results?buildId=3072363&view=results). | Watch — cause unconfirmed this run | latest finished ~47m ago |
| 🟡 | One PR turns package vulnerability warnings into build errors across many jobs after exhausting four attempts ([run](https://github.com/microsoft/aspire/actions/runs/34526846888)). | PR author owns it; the rerun bot is already trying the newer PR tip | last completed ~1.5h ago |

🧱 Product vs infra: rolling `main` is blocked by the FluentUI package signature and a Homebrew cask incompatibility, while outerloop still has four reproducible product/test failures after all retries. The release failures look flaky or environmental, internal causes remain unconfirmed, and the latest completed PR failure is isolated to that PR's audit configuration.

⏱️ Rolling `main` is also slower today: build-time p50 is ~52m versus ~36m over the 7-day baseline.

exact tests, jobs & counts

**🔴 Rolling `GH CI — main` package restore** — 1/12 completed runs passed in the 36h window. The FluentUI failure reproduced in four consecutive pushes:

- Latest run [`34519184641`](https://github.com/microsoft/aspire/actions/runs/34519184641), commit `9c2af96f`: `Aspire.Dashboard.Tests.csproj` failed with `NU3027` because `Microsoft.FluentUI.AspNetCore.Components 5.0.0-preview.26251.2` was not timestamp-signed for long-term validity. The restore also reported a missing dictionary key for `SQLitePCLRaw.core 3.0.5`. The FluentUI version is set at [`Directory.Packages.props:10`](https://github.com/microsoft/aspire/blob/9c2af96f9163e2d89c2e9e789f5d40c395606f14/Directory.Packages.props#L10).
- Run [`34519005883`](https://github.com/microsoft/aspire/actions/runs/34519005883) hit the same `NU3027` failure in `Aspire.Hosting.RemoteHost.Tests.csproj`.
- Run [`34512081344`](https://github.com/microsoft/aspire/actions/runs/34512081344) hit the same `NU3027` failure in `Aspire.Hosting.RemoteHost.Tests.csproj`; its restore also saw one transient Azure DevOps `503`.
- Run [`34508764231`](https://github.com/microsoft/aspire/actions/runs/34508764231) failed on macOS and Linux with `NU1100` for the FluentUI component and icon versions and on Windows with the same `NU3027` signature rejection.
- The four triggering commits changed unrelated workflow, test, Front Door, or dev-tunnel areas, strengthening the package-feed/signature diagnosis.

**🔴 Rolling `GH CI — main` Homebrew validation** — cask preparation failed in 2 of the last 4 pushes:

- Runs [`34519184641`](https://github.com/microsoft/aspire/actions/runs/34519184641) and [`34512081344`](https://github.com/microsoft/aspire/actions/runs/34512081344) both failed `brew audit --except=installed --tap=local/aspire`.
- Runs [`34519005883`](https://github.com/microsoft/aspire/actions/runs/34519005883) and [`34508764231`](https://github.com/microsoft/aspire/actions/runs/34508764231) passed the same job, so this is intermittent while runner Homebrew versions converge, not deterministic.
- Verbatim error: `the 'verified' parameter has been deprecated; use the 'url' stanza without it`.
- The generated stanza comes from [`eng/homebrew/aspire.rb.template:8-9`](https://github.com/microsoft/aspire/blob/9c2af96f9163e2d89c2e9e789f5d40c395606f14/eng/homebrew/aspire.rb.template#L8-L9).

**🔴 Outerloop dashboard** — attempt 4 failed **40/53 tests on Linux and 40/53 on Windows**; both dashboard jobs failed in attempts 1, 2, 3, and 4.

- Representative timeout: `Aspire.Dashboard.Tests.Integration.Playwright.AppBarTests.AppBar_Change_Theme` timed out after 30 seconds at [`AppBarTests.cs:24-29`](https://github.com/microsoft/aspire/blob/b8ac90b2b84deb9d124ee51ce721c5c02d48dae7/tests/Aspire.Dashboard.Tests/Integration/Playwright/AppBarTests.cs#L24-L29).
- Other failures span app-bar layout, authentication redirects, resource menus, accessibility surfaces, focus indicators, and viewport checks on both OSes. Representative errors include `Locator expected to be visible`, `No control part found for fluent-button`, and 30-second operation timeouts.

**🔴 Outerloop Maven/JDK** — `Aspire.Cli.Tests.Projects.JavaAppHostToolchainResolverTests.ApplyToRuntimeSpec_ForMavenReactor_StagesAnUninstalledSiblingFromAnEmptyLocalRepository` failed 1/13 tests on Linux and 1/13 on Windows in attempt 4; both CLI jobs failed in all four attempts.

- Verbatim error: `Fatal error compiling: error: release version 21 not supported`.
- Assertion: [`JavaAppHostToolchainResolverTests.cs:406`](https://github.com/microsoft/aspire/blob/b8ac90b2b84deb9d124ee51ce721c5c02d48dae7/tests/Aspire.Cli.Tests/Projects/JavaAppHostToolchainResolverTests.cs#L406).

**🔴 Outerloop agent initialization** — 3/15 CLI end-to-end tests failed in attempt 4; the job failed in all four attempts.

- Tests: `AspireNew_WithAgentInit_InstallsPlaywrightWithoutErrors`, `AgentInit_InstallsPlaywrightCli_AndGeneratesSkillFiles`, and `AgentInit_CwdDiffersFromRoot_PlacesSkillsInWorkspaceRoot`.
- Verbatim error: `Provenance verification failed for @playwright/cli@0.1.19: AttestationFetchFailed`.
- Assertions: [`NewWithAgentInitTests.cs:112-125`](https://github.com/microsoft/aspire/blob/b8ac90b2b84deb9d124ee51ce721c5c02d48dae7/tests/Aspire.Cli.EndToEnd.Tests/NewWithAgentInitTests.cs#L112-L125), [`PlaywrightCliInstallTests.cs:55-61`](https://github.com/microsoft/aspire/blob/b8ac90b2b84deb9d124ee51ce721c5c02d48dae7/tests/Aspire.Cli.EndToEnd.Tests/PlaywrightCliInstallTests.cs#L55-L61), and [`PlaywrightCliInstallTests.cs:110-116`](https://github.com/microsoft/aspire/blob/b8ac90b2b84deb9d124ee51ce721c5c02d48dae7/tests/Aspire.Cli.EndToEnd.Tests/PlaywrightCliInstallTests.cs#L110-L116). Tracking issue: [#19159](https://github.com/microsoft/aspire/issues/19159).

**🔴 Outerloop Redis starter template** — all 3 selected tests failed in attempt 4; the templates job failed in all four attempts.

- Tests: `ResourcesShowUpOnDashboard` and both `ApiServiceWorks` variants on `StarterTemplateWithRedisCacheTests`.
- Verbatim error: `Expected rows not found: apiservice, webfrontend, cache`.
- Assertion: [`TemplateTestsBase.cs:162-257`](https://github.com/microsoft/aspire/blob/b8ac90b2b84deb9d124ee51ce721c5c02d48dae7/tests/Aspire.Templates.Tests/TemplateTestsBase.cs#L162-L257).

**🟡 `release/14.0` Windows Kafka job** — the [job](https://github.com/microsoft/aspire/actions/runs/34508166356/job/102976255265) reported `Passed!` for all 492 tests (474 succeeded, 18 skipped, 0 failed), then three post-test commands exited `-1073741502`. This is one run with no product-test failure, so it is too early to call a recurring break.

**🟡 `release/14.0` Deno debugger E2E** — the [job](https://github.com/microsoft/aspire/actions/runs/34508166356/job/102980021974) had 0 passing and 1 failing test:

- `attaches js-debug to a Deno resource and hits a TypeScript breakpoint` timed out after 240000 ms waiting for `main.ts:2`, with `Last error: `.
- Test setup and assertions: [`denoDebugger.e2e.test.ts:24-76`](https://github.com/microsoft/aspire/blob/66f5aaa3930fb3f9e5063d5062e046569bb3a647/extension/src/test-e2e/denoDebugger.e2e.test.ts#L24-L76).
- An earlier ChromeDriver metadata request hit `ECONNRESET` but succeeded on retry, so that recovered download is not the final test failure.

**🟡 Internal (AzDO)** — `main` had 0 succeeded or partially succeeded builds out of 9 completed in-window. Builds `3071054`, `3071262`, `3071391`, `3071497`, `3071585`, `3071656`, `3072001`, `3072274`, and `3072363` failed; build `3070652` immediately before the window makes ten consecutive failures. Aggregate counts only per internal-privacy policy; causes were not inspected.

**🟡 PR validation** — 39/73 counted runs were red in the 36h window. The latest completed run, [`34526846888`](https://github.com/microsoft/aspire/actions/runs/34526846888) for [#20052](https://github.com/microsoft/aspire/pull/20052), exhausted four attempts because `NU1902`/`NU1903` package-vulnerability warnings became errors across many jobs. A newer PR run is currently in progress on its second bot-triggered attempt.

_{Attention dots: 🔴 needs a human now · 🟡 keep an eye on it · ⚪ looks resolved. These are **distinct** from the lane table's 🔴/🟢 run-outcome dots.}

### 🔁 Reruns & flaky tax

The **auto-rerun bot carried 147 of 154 reruns** and rescued 17 PR runs. Human-only reruns rescued none, so PR flakiness remains overwhelmingly automated toil with manual tax at the edge.

rerun actor split (36h)

🤖 **Auto-rerun bot:** 147 reruns across 62 PR runs → **17 rescued** red→green; 36 still red, 9 cancelled.

🧑 **Manual (human) reruns:** 7 reruns across 3 PR runs (1 branches the bot also reran) → on the 2 human-only branches: **0 rescued**, 2 still red, 0 cancelled.

🔗 **Combined: 65 distinct PR runs** needed a rerun in the window (18 went green, 38 still red, 9 cancelled); **1 PR branches** needed both bot + human.

_{Sources: `gh api .../ci.yml/runs` (pass rates + rerun split, fetched in bounded UTC-day buckets), `az pipelines build list` def 1602 (internal, aggregate counts only), and failed GitHub job logs for live-cause confirmation. **ciinsights was unavailable in this session**, blocking fresh failure-cluster and infra-share joins; all visible causes were confirmed directly from GitHub job data and logs. Window = last 36h. Backlog & trend: see the weekly CI Health report (#18231). Generated Sep 10, 2026 21:55 UTC.}