[AGR] Track agent pipeline effectiveness metrics over time

# Track agent pipeline effectiveness metrics over time

> **Status as of 2026-06-17:** This issue should become the live metrics and observability source for the agentic workflow under #32454. Raw `s/agent-*` label counts are operational signals only until label semantics, transitions, and time-in-state are audited. It consumes structured data from the shipped scanners/workflows below; it does not produce new CI evidence (that's #35052).

## Problem

The repository has meaningful agent activity across issue triage, PR review, CI analysis, and release-readiness work. However, there is no trusted trend view that answers whether the system is getting more reliable over time.

Questions we need to answer:

- How long do PRs spend in each `s/agent-*` state?
- Which gates fail most often, and why?
- How often does a test fail before the fix and pass after the fix?
- How often does CI classification identify a known/flaky failure vs a suspected product regression?
- How often do agent-reviewed PRs merge without human rework?
- How often do regression safeguards catch issues before merge?
- Which release-readiness blockers recur across release trains?

## Metrics principles

- Do not treat label counts alone as quality metrics.
- Track time-in-state, transitions, and outcomes.
- Separate throughput metrics from correctness metrics.
- Keep metrics read-only until semantics are audited.
- Tie every aggregate number back to reproducible GitHub/AzDO/Helix evidence.
- Consume #35052's CI record rather than re-parsing GitHub check summaries; preserve its four classification axes (do not collapse them).

## Workflow data sources (shipped)

The metrics view reads from these shipped/scheduled workflows. It does **not** expand their scope.

| Workflow | Introduced by | Surface (current) |
|----------|---------------|-------------------|
| `.github/workflows/ci-status-main.md` | #35734 (merged 2026-06-08) | Emits **`[ci-scan]`** issues (per recurring failure-signature) for failing `maui-pr` / `maui-pr-devicetests` / `maui-pr-uitests` builds on `main`, each with a `**Branch**: main` marker. **Human-readable markdown today;** the `ci-status-json` machine block is a planned #35052 deliverable. |
| `.github/workflows/ci-status-net11.md` | #35734 (merged 2026-06-08) | Emits **`[ci-scan-net11]`** issues for `net11.0` builds (same shape + marker). ~26 `[ci-scan]`/`[ci-scan-net11]` issues across both scanners in the first ~7 days. |
| `.github/workflows/rerun-review-scanner.md` | #35685 (merged) | Summary issue per completed `/review` rerun event. |
| `.github/workflows/copilot-review-tests.md` (+ shared `/review tests` infra) | #35701 (merged) + #35820 / #35817 (merged 2026-06-09) | Triggered review-tests output and rerun-summary fixes. |
| `.github/workflows/daily-repo-status.md` | (existing) | Daily snapshot of PR/issue queues, agent labels, repo health (e.g. #35698 / #35699). |
| `.github/workflows/ci-status-fix.md` | #35927 (draft) | "CI Failure Fixer" — walks `[ci-scan]` issues and opens draft `[ci-fix]` PRs (capped at 5 attempts → one `[ci-fix][needs-human]` hand-off). Owned by #35978. Source for the CI auto-fix loop metrics below. |
| `.github/workflows/regression-corpus-scanner.md` | #35925 (draft) | Mines merged `i/regression` fixes → authors a hermetic `code-review` eval stimulus as one draft test-only PR. Source for the "auto-added eval fixtures" count. |

Generated daily report issues (e.g. #35698, #35699) are **inputs**, not durable trackers.

## Metrics to track

| Category | Examples |
|----------|----------|
| Lifecycle | time in `s/agent-ready`, `s/agent-reviewed`, `s/agent-gate-passed`, `s/agent-fix-win`, `s/agent-changes-requested` |
| Gate outcomes | reproduction gate pass/fail, fix gate pass/fail, test verification failures |
| Review outcomes | AI summary disposition, inline findings, re-review/rerun count, human follow-up required |
| CI truth (per-build, from #35052) | four-axis classification — failure type / cause (`product-regression-suspected`, not `real-regression`) / relation-to-change / known-issue state; per-build `status`; KBE match + stale-KBE rate |
| Regression prevention | regression-risk findings, escaped regressions, eval fixtures added from misses (incl. corpus entries auto-drafted by #35925) |
| CI auto-fix loop (from #35978) | `[ci-fix]` attempt count per `[ci-scan]` issue (capped 5), land vs closed-unmerged rate, `[ci-fix][needs-human]` hand-off rate, "appears-fixed-in-latest-build" skip rate; share of `[ci-scan]` issues worked by humans vs the fixer (e.g. #35969) |
| Issue triage | area/platform label accuracy, missing-info precision, regression detection recall, ready-gate false positives |
| Release health (per-target, from #35711) | Maestro PR age/blocking reason, release blocker counts, KBE release impact, per-target `releaseStatus` (distinct from #35052's per-build `status`) |
| Merge outcomes | merge rate, time-to-merge, author rework after agent review |

## Data sources

- GitHub labels, issue/PR timelines, comments, reviews, and milestones
- The shipped workflows above (consume `[ci-scan]` issues via the `**Branch**:` marker today; the `ci-status-json` block when #35052 ships it)
- #35048 PR review output
- #35051 triage workflow output
- #35052 CI truth/classification output
- #35978 CI Failure Fixer loop (`[ci-fix]` PRs + their `Refs: dotnet/maui#` back-links) and #35925 regression-corpus PRs
- #34603 issue-to-PR gate evidence
- #35711 release-readiness output (live Release READY issues #35866 / #35867 / #35876)
- AzDO/Helix evidence where needed for CI metrics

## Acceptance criteria

- [ ] Define canonical semantics for every `s/agent-*` label used in the repo.
- [ ] Define valid state transitions and terminal states.
- [ ] Produce a weekly report of lifecycle counts and time-in-state.
- [ ] Track gate pass/fail outcomes separately from merge outcomes.
- [ ] Track CI classifications from #35052, preserving the four axes (do not collapse them); consume the record rather than re-parsing GitHub check summaries (a missing/stale machine block counts as `INSUFFICIENT_DATA`).
- [ ] Report per-build CI `status` (#35052) and per-target `releaseStatus` (#35711) as distinct metrics, not one merged "READY" number.
- [ ] Track issue triage accuracy and ready-gate false positives from #35051.
- [ ] Track release-health signals from #35711.
- [ ] Track the CI auto-fix loop from #35978 (`[ci-fix]` attempt/land/hand-off rates) and auto-added eval-fixture counts from #35925.
- [ ] Flag metric regressions after workflow/skill changes.
- [ ] Avoid using historical one-time snapshots as live KPIs.
- [ ] Add no new scanners (consume the workflows above).

## Related

- Parent epic: #32454
- Issue triage: #35051
- Issue-to-PR pipeline: #34603
- PR review: #35048
- CI truth: #35052
- CI Failure Fixer loop: #35978
- Skill eval lifecycle: #34814
- Release readiness: #35711

What changed in this revision (vs the 2026-06-02 body)

Additive — keeps the Problem, Metrics principles, Metrics-to-track table, Data sources, and Related sections. Changes:

- Status date 2026-06-02 → 2026-06-17.
- Added a **Workflow data sources (shipped)** table (#35734 `[ci-scan]`/`[ci-scan-net11]`, #35685, #35701/#35820/#35817, daily-repo-status) with the real label + `**Branch**:` marker mechanism, noting the JSON machine block is a planned #35052 deliverable.
- Metrics principles + CI-truth row: reframed around #35052's **four separable axes**; replaced `real-regression` with `product-regression-suspected`.
- Distinguished per-build CI `status` (#35052) from per-target `releaseStatus` (#35711); pointed Release-health at the live issues #35866/67/76.
- Acceptance criteria: consume the record (not check summaries) with an `INSUFFICIENT_DATA` fallback; keep the two rollups distinct; preserve the four axes; add no new scanners.

**2026-06-17 follow-up:** added the new detect→fix and regression-corpus workflows as **sources** (not new scanners) — `ci-status-fix.md` (#35927, owned by #35978) and `regression-corpus-scanner.md` (#35925) — plus a "CI auto-fix loop" metrics row and an "auto-added eval fixtures" signal.

Start by reading the shipped workflow definitions under .github/workflows, especially ci-status-main.md, ci-status-net11.md, daily-repo-status.md, and the CI fixer and regression-corpus sources. Review outputs from #35051, #35052, #35711, #35978, and #35925, then define the canonical states, transitions, and weekly report described in the acceptance criteria without adding scanners.

Written by the indexing model from the issue text.