Automated & scheduled workflow health — rolling 9-day dashboard

# Automated & scheduled workflow health

Read-only projection rendered from the monitor's store. Do not edit by hand.

_3 of 18 tracked workflows are red on their latest run: CI on the release branch, outerloop tests, and deploy E2E. PR Documentation Check is green in Actions but its agentic side-path keeps filing failure issues._

## 🔴 Needs attention — currently broken

| Workflow | Latest run | What's broken |
|---|---|---|
| 🟡 GH CI — release/13.4 | ❌ [failure](https://github.com/microsoft/aspire/actions/runs/28005419455) (06-23 05:53) | The `Hosting-1 (windows-latest)` test leg timed out — a hang dump was detected and the run gate exited 1. `release/13.4` has had no newer rolling CI run since, so latest stays red. [9-day trend ↓](#h-ci-rel) |
| 🟡 Outerloop Tests | ❌ [failure](https://github.com/microsoft/aspire/actions/runs/28214555464) (06-26 03:07) | `tests/Aspire.Cli.Tests/Telemetry/TelemetryConfigurationTests.cs` failed to compile (`error CS1061: 'Task' does not contain a definition for 'DefaultTimeout'`), so the runsheet build aborted. Already fixed on `main` by #18454 (merged 06-26 19:02); awaiting the next scheduled run. [9-day trend ↓](#h-outerloop) |
| 🧑 Deployment E2E Tests | ❌ [failure](https://github.com/microsoft/aspire/actions/runs/28217197162) (06-26 04:29) | Build hit the **same** `CS1061 DefaultTimeout` compile break (fixed by #18454); underneath it the AKS deploy E2E is still failing and is tracked by #18423. [9-day trend ↓](#h-deploy-e2e) |
| 🟡 PR Documentation Check | ✅ [success](https://github.com/microsoft/aspire/actions/runs/28260599211) (06-26 19:29) | Actions runs are green, but the agentic check keeps filing `[aw]` failure / incomplete-result issues (latest #18509, 06-26). Deduped this run to #18509 + #18504 (labeled `automation-broken`); older duplicates closed. [9-day trend ↓](#h-pr-docs) |

Legend: 🔴 broken · 🟡 worth a glance / recovering · 🧑 human-only (not AI-fixable) · 🟢 healthy.

Not listed (not broken now): **Internal — main**, **Internal — release/13.4**, **GH CI — main**, **Deployment Environment Cleanup**, **Daily CLI Smoke Tests**, **Warm CLI E2E Image Cache**, **Quarantined Tests**, **Generate API Diffs**, **Generate ATS Diffs**, **Refresh Manifests**, **Update Microsoft Foundry Models**, **Update Aspire Skills Bundle**, **Update Dependencies**, **Update Azure VM Sizes** — all green on their latest run. Bot / auto-opened PRs and issues are shepherded by #18285.

## Orchestrator queue

_No unworked AI-fixable freeform break this run. The two compile-error breaks (Outerloop, Deploy E2E) already have a merged fix (#18454); GH CI release leg is an isolated flaky/timeout leg; PR Documentation Check is tracked by #18509 / #18504 — all kept out of the fence._

## 9-day health

How each workflow has trended over the last 9 days, independent of the latest run above. **Pass / Runs** counts only *completed* runs (success or failure); cancelled / skipped / in-progress are excluded. Emoji = 9-day pass rate (🟢 ≥80% · 🟡 40–79% · 🔴 <40% · ⚪ no runs).

| Workflow | Pass / Runs (9d) | Health note |
|---|---|---|
| Internal — main | 🟢 **70 / 79 — 89%** | Steady; latest build green. |
| Internal — release/13.4 | 🟡 **3 / 4 — 75%** | Low run count; latest build green. |
| GH CI — main | 🔴 **39 / 99 — 39%** | Chronically flaky over the window; latest run green. |
| GH CI — release/13.4 | 🟡 **2 / 3 — 67%** | Latest run red — Hosting-1 (windows) leg timed out; see Needs-attention. |
| Outerloop Tests | 🟡 **5 / 9 — 56%** | Latest run red on a compile break (fix #18454 merged); see Needs-attention. |
| Deployment E2E Tests | 🔴 **0 / 9 — 0%** | Red every night; AKS deploy E2E tracked by #18423; see Needs-attention. |
| Deployment Environment Cleanup | 🟢 **155 / 155 — 100%** | Hourly infra signal clean; latest run green. |
| Daily CLI Smoke Tests | 🟢 **9 / 9 — 100%** | Clean; latest run green. |
| Warm CLI E2E Image Cache | 🟢 **9 / 9 — 100%** | Cache warm; latest run green. |
| Quarantined Tests | 🟡 **81 / 106 — 76%** | Flaky shards trip whole runs; latest run green. |
| PR Documentation Check | 🟢 **71 / 80 — 89%** | Actions runs green; agentic side-path filing `[aw]` issues — see Needs-attention. |
| Generate API Diffs | 🟢 **9 / 9 — 100%** | Clean; latest run green. |
| Generate ATS Diffs | 🟢 **9 / 9 — 100%** | Clean; latest run green. |
| Refresh Manifests | 🟡 **5 / 9 — 56%** | Recovered — red 06-19→06-22, green since; latest run green. |
| Update Microsoft Foundry Models | 🟢 **9 / 9 — 100%** | Clean; latest run green. |
| Update Aspire Skills Bundle | 🟢 **10 / 10 — 100%** | Clean; latest run green. |
| Update Dependencies | 🟢 **1 / 1 — 100%** | Weekly (Mon); latest run green. |
| Update Azure VM Sizes | 🟢 **3 / 3 — 100%** | Switched to daily on 06-24 and now green (was the monthly failure in #18305). |

📈 Per-day trend — most recent first (emoji = that day's pass%)

| Day | Internal — main | Internal — release/13.4 | GH CI — main | GH CI — release/13.4 | Outerloop Tests | Deployment E2E Tests | Deployment Environment Cleanup | Daily CLI Smoke Tests | Warm CLI E2E Image Cache | Quarantined Tests | PR Documentation Check | Generate API Diffs | Generate ATS Diffs | Refresh Manifests | Update Microsoft Foundry Models | Update Aspire Skills Bundle | Update Dependencies | Update Azure VM Sizes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2026-06-26 | 🟢 | — | 🟡 | — | 🔴 | 🔴 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | — | 🟢 |
| 2026-06-25 | 🟢 | — | 🔴 | — | 🔴 | 🔴 | 🟢 | 🟢 | 🟢 | 🔴 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | — | 🟢 |
| 2026-06-24 | 🟢 | — | 🟡 | — | 🔴 | — | 🟢 | — | — | 🟢 | 🟢 | — | 🟢 | — | — | — | — | 🟢 |
| 2026-06-23 | 🟢 | 🟡 | 🟡 | 🔴 | 🟢 | 🔴 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | — | — |
| 2026-06-22 | 🟢 | — | 🔴 | — | 🟢 | 🔴 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🔴 | 🟢 | 🟢 | 🟢 | — |
| 2026-06-21 | 🟡 | — | 🟢 | — | 🟢 | 🔴 | 🟢 | 🟢 | 🟢 | 🟢 | — | 🟢 | 🟢 | 🔴 | 🟢 | 🟢 | — | — |
| 2026-06-20 | 🟢 | — | 🟡 | — | 🟢 | 🔴 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | 🔴 | 🟢 | 🟢 | — | — |
| 2026-06-19 | 🟢 | 🟢 | 🟡 | 🟢 | 🟢 | 🔴 | 🟢 | 🟢 | 🟢 | 🟡 | 🟢 | 🟢 | 🟢 | 🔴 | 🟢 | 🟢 | — | — |
| 2026-06-18 | 🟢 | 🟢 | 🔴 | — | 🔴 | 🔴 | 🟢 | 🟢 | 🟢 | 🟡 | 🟡 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 | — | — |

## Per-workflow detail

- 🟢 **Internal — main** — latest build green; 89% over the window (70/79).
- 🟡 **Internal — release/13.4** — latest build green; 75% (3/4), low run count.
- 🔴 **GH CI — main** — latest run green; chronically flaky at 39% (39/99) over the window.
- 🟡 **GH CI — release/13.4** — latest run [28005419455](https://github.com/microsoft/aspire/actions/runs/28005419455) red: `Hosting-1 (windows-latest)` leg timed out (hang dump). No newer release CI run since 06-23. 67% (2/3).
- 🟡 **Outerloop Tests** — latest run [28214555464](https://github.com/microsoft/aspire/actions/runs/28214555464) red: `TelemetryConfigurationTests.cs` compile break (`CS1061 DefaultTimeout`), introduced by #18251 and fixed by #18454 (merged 06-26 19:02). 56% (5/9).
- 🧑 **Deployment E2E Tests** — latest run [28217197162](https://github.com/microsoft/aspire/actions/runs/28217197162) red: same `CS1061 DefaultTimeout` build break (fixed by #18454); the AKS deploy E2E underneath stays red, tracked by #18423. 0% (0/9).
- 🟢 **Deployment Environment Cleanup** — latest run green; 100% (155/155). Hourly deploy-infra signal clean.
- 🟢 **Daily CLI Smoke Tests** — latest run green; 100% (9/9).
- 🟢 **Warm CLI E2E Image Cache** — latest run green; 100% (9/9).
- 🟡 **Quarantined Tests** — latest run green; 76% (81/106). Red runs are flaky shards tripping the whole run (known by-design bug); not real workflow breaks.
- 🟡 **PR Documentation Check** — Actions runs green (89%, 71/80), but the agentic check keeps filing `[aw]` failure / incomplete issues. Deduped to #18509 (incomplete) + #18504 (failed), labeled `automation-broken`; older duplicates #18478 / #18441 / #18434 / #18408 / #18397 closed.
- 🟢 **Generate API Diffs** — latest run green; 100% (9/9).
- 🟢 **Generate ATS Diffs** — latest run green; 100% (9/9).
- 🟢 **Refresh Manifests** — latest run green; 56% (5/9) — red 06-19→06-22, green since.
- 🟢 **Update Microsoft Foundry Models** — latest run green; 100% (9/9).
- 🟢 **Update Aspire Skills Bundle** — latest run green; 100% (10/10).
- 🟢 **Update Dependencies** — latest run green; 100% (1/1). Weekly (Mondays).
- 🟢 **Update Azure VM Sizes** — latest run green; 100% (3/3). Schedule changed to daily on 06-24 (`cron '0 6 * * *'`); the prior monthly failures (#18305) no longer recur.

_{Report generated 2026-06-26 21:10 UTC · rolling 9-day window (2026-06-18 → 2026-06-26).}