dotnet / dotnet/maui

Autonomous CI Failure Fixer — turn [ci-scan] issues into [ci-fix] PRs

Open
#35,978 0 comments 1 reaction 0 assignees View on GitHub
area-infrastructure s/triaged
Dominant language
C#
Stars
23.3k
Forks
2k
Avg merge
1d 14h
Merged PRs (30d)
296

Description

> **Status as of 2026-06-17:** New sub-issue under #32454, Pillar 3 (CI & PR Review). It owns the **fix** half of the CI detect→fix loop. The **detect** half (the `[ci-scan]` scanners and their output contract) is owned by #35052; this issue consumes that output and does not duplicate or re-own it. Sibling of #35052 (CI truth/detection) and #35048 (PR review). The implementation is **in-flight as draft PR #35927** — nothing here is shipped yet.

## Context

The CI analysis story now has two halves:

- **Detect** (shipped, #35734; owned by #35052): `.github/workflows/ci-status-main.md` / `ci-status-net11.md` emit `[ci-scan]` / `[ci-scan-net11]` tracking issues — one per recurring failure-signature — each carrying a `**Branch**:` marker. As of 2026-06-17 there are ~31 open (`16` `[ci-scan]` on `main`, `15` `[ci-scan-net11]`).
- **Fix** (this issue; in-flight as draft #35927): `.github/workflows/ci-status-fix.md` ("CI Failure Fixer") walks those `[ci-scan]` issues and opens **draft `[ci-fix]` PRs** against the matching branch.

Until #35927, the `[ci-scan]` queue had no automated remediation path — issues accumulated and were worked only by humans (e.g. #35969, a Syncfusion partner PR addressing `[ci-scan-net11]` #35743). This issue tracks the autonomous fixer that proposes a first-pass fix and a durable human hand-off when it can't.

## Ownership split

- **This issue owns:** the fix executor — walking `[ci-scan]` issues, opening draft `[ci-fix]` PRs, attempt/retry/hand-off lifecycle, and the safety bounds on what an autonomous fix may touch.
- **#35052 owns:** detection, classification, and the **scanner output contract** — including the `Build ID: ` line and `` marker that #35927 adds to the scanner templates. Those edits change scanner *output format* and are reviewed as #35052 acceptance items even though they ride in on #35927's PR. The future machine-readable `ci-status-json` record is also #35052's.
- **#34603 owns:** the autonomous **issue→PR pipeline for agent-candidate product bugs**, whose hard invariant is *"no PR without a confirmed failing pre-fix reproduction test."* This fixer is a **sibling, not a child**, of #34603: its gate is structurally different — *"the failure signature still reproduces in the latest CI build on the target branch"* — not a newly authored pre-fix repro test. The two pipelines must not be collapsed; merging their gates would muddy #34603's contract.
- **#35711 consumes** the result as a pipeline-health signal (green pipelines → release readiness). It does not run fixes.
- **#34604 owns** the metrics for this loop (attempt counts, land rate, hand-off rate — see below).

## How it works (design highlights, per #35927)

- **Branch-aware at four layers.** `[ci-scan]` → `main`, `[ci-scan-net11]` → `net11.0`. The base is enforced by (1) a gh-aw declarative gate (`allowed-base-branches: [main, net11.0]`), (2) a prompt rule deriving the branch from the issue label (never body content), (3) a post-checkout filesystem assertion, and (4) a self-check grep of its own PR body before emission. Any mismatch aborts.
- **Iterative, capped at 5 attempts per tracking issue.** Attempt count is derived from a live GitHub search for prior closed-unmerged `[ci-fix]` PRs referencing the issue (`Refs: dotnet/maui#`) — GitHub is the durable store, no per-run state. The 6th tick opens **one** `[ci-fix][needs-human]` hand-off PR and never retries. Each attempt must read prior approaches/close-comments and propose a substantively different one.
- **"Is it actually fixed?" pre-check.** Before attempting, the agent fetches the latest completed build of the failing pipeline on the target branch and greps the failure signature against the leaf log. Zero hits → skip ("appears fixed"); tracking-issue closure stays a human decision.
- **Visual-regression skip gate (first gate).** Any issue whose signature matches `screenshot` / `snapshot` / `visual diff` / `baseline image` / `VerifyScreenshot` / etc. is skipped — gh-aw can't judge visual diffs and must never touch baseline images.
- **Never mutes a test.** Adding `[ActiveIssue]` / `Skip` / `[SkipOnPlatform]`, csproj `<*Incompatible>` / ``, or editing baseline images is detected and rejected. If a mute is the only candidate fix, it records a skip and stops.
- **MAUI area bounds.** Small compile/XAML breaks in `maui-pr` are in bounds (≤20 lines, single file when possible). Device/UI-test failures past the visual gate become `help`-only PRs (`Validation: not run`). Handler lifecycle, threading, safe-area, perf hot-paths, Gradle/Maven feed, and infra failures are skipped as too risky for an autonomous fix.
- **Outputs only via `safe-outputs`.** `[ci-scan]` issues are locked (no comments). `draft: true`, one PR per run (`max: 1`), branch restricted to `ci-fix/**`. Writes are confined by an `allowed-files` allowlist to `src/Core|Controls|Essentials|BlazorWebView|TestUtils|Templates/**` + `**/PublicAPI.Unshipped.txt`, so `.github/**` (and anything off the list) cannot be edited; `protect-top-level-dot-folders` additionally guards top-level dot folders, and a `protected-files` list (dependency manifests such as `global.json` / `NuGet.Config` / `Directory.Packages.props` and key docs) is gated behind `request_review`. Does **not** read PR review comments as authoring input.

## Scope

- The `ci-status-fix.md` workflow and its lifecycle/stop-conditions.
- The `[ci-fix]` / `[ci-fix][needs-human]` PR conventions and the `Refs: dotnet/maui#` back-link.
- The safety bounds above (branch enforcement, no-mute, visual-skip, area bounds, draft-only).
- Coordinating with #35052 on the two scanner-contract fields the fixer relies on (`Build ID`, `ci-scan-match-count`).

## Non-goals

- Do **not** own the scanner or its output contract (that's #35052).
- Do **not** mute, skip, or `[ActiveIssue]` a test, or modify visual baselines.
- Do **not** close `[ci-scan]` tracking issues — closure is a human decision.
- Do **not** integrate KBE / Build Analysis (explicitly out of scope for the first cut).
- Do **not** auto-merge; output is always a draft for human review.
- Do **not** treat PR review comments as authoring instructions.
- Do **not** absorb #34603's pre-fix-reproduction gate — keep the two pipelines' gates separate.

## Acceptance criteria

- [ ] `ci-status-fix.md` opens at most one draft `[ci-fix]` PR per `[ci-scan]` issue per run, on the branch derived from the issue's label.
- [ ] Attempts are capped at 5 per issue; the 6th produces exactly one `[ci-fix][needs-human]` hand-off PR and no further retries.
- [ ] The "appears fixed in latest build" pre-check skips issues whose signature no longer reproduces.
- [ ] The visual-regression gate and never-mute guard are enforced and covered by evals (#34814).
- [ ] Every emitted PR carries a `Refs: dotnet/maui#` back-link to its `[ci-scan]` issue.
- [ ] The fixer relies only on the `Build ID` / `ci-scan-match-count` scanner fields owned by #35052 — no re-parsing of GitHub check summaries.
- [ ] Loop metrics (attempt count, land vs closed-unmerged rate, `[needs-human]` rate, "appears-fixed" skip rate) are reported in #34604.
- [ ] Tracking-issue closure remains a human action.

## Current state

- **#35927** — draft, opens the `ci-status-fix.md` workflow + the two scanner-contract edits. Currently test-running from its own branch.
- **#35976** — `[aw] CI Failure Fixer failed`: a transient `copilot` engine termination on one run (infrastructure flake, not a logic defect). Confirms the workflow is already executing live from the PR branch.
- No `[ci-fix]` PR authored by this workflow has landed yet; the `[ci-scan]` queue is also being worked by humans in parallel (#35969).

## Related

- Parent epic: #32454 (Pillar 3)
- CI truth / detection + scanner contract: #35052
- PR review pipeline: #35048
- Autonomous issue→PR for product bugs (sibling, different gate): #34603
- Release/pipeline-health consumer: #35711
- Metrics: #34604
- Eval coverage for the fixer's safety gates: #34814
- Implementation PR: #35927

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/ci-status-fix.md and draft PR #35927, then inspect the safety-gate evals tracked in #34814. Done means the workflow enforces the stated branch, retry, visual-regression, no-mute, draft-only, and backlink rules while producing the required hand-off behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.