dotnet / dotnet/sdk

Agentic workflow: detect build breaks and intermittent test failures and open tracking issues

Open
#55,553 3 comments 0 reactions 1 assignee Claimed by @nagilson View on GitHub
Area-Infrastructure
Dominant language
C#
Stars
3.2k
Forks
1.3k
PR merge metrics
PR metrics pending

Description

# Agentic workflow: detect build breaks and intermittent test failures and open tracking issues

## Summary

We need an **agentic workflow** in the .NET SDK repo that continuously watches our **public** CI (builds and PRs) and automatically opens issues for CI failures, so that build breaks and flakiness are tracked aggressively and consistently rather than depending on someone noticing and filing by hand. It should handle two cases:

1. **Build breaks** (hard failures) — open a tracking issue with the details of the break.
2. **Intermittent failures** (flaky tests / non-deterministic build failures) — open a **Known Build Error (KBE)** issue so Build Analysis can match future occurrences and retry when appropriate.

## Motivation

Our CI success rate degrades over time without active tracking of build breaks and known build errors. We recently put real effort into driving the pass rate back up, but history shows it regresses again unless this tracking is automated. Doing it manually — finding the failure, checking whether an issue already exists, authoring a correct KBE pattern, deciding whether retry is appropriate — is slow, inconsistent, and dependent on an individual remembering to do it.

Automating it keeps our failure tracking current and our Build Analysis coverage honest, reduces triage toil, and increases team productivity by surfacing real breaks quickly and getting flakiness on the books with an owner.

## Desired behavior (high level)

- **Monitor public CI** — builds and PRs for the SDK repo. In particular, catch failures that get masked before merge, e.g. PRs that used `/ba-g` or that required a re-run / retry before going green, as well as failures on rolling/main builds.

- **Always check for existing issues first.** Before opening anything, look for an already-tracking issue (open *or* closed) for the same failure and avoid filing duplicates.

- **For build breaks:** open a tracking issue capturing the failure — the error/stack, the failing leg, and links to the build(s) and the offending PR/commit where relevant.

- **For intermittent failures:** open a **Known Build Error** issue. This should use the **Arcade skill for Known Build Errors** — there is an existing Arcade skill that knows how to author a KBE issue, validate the regex error pattern, and decide whether retry is appropriate, and it should be leveraged here rather than re-implementing that logic. The resulting issue should include a **validated error pattern** that is specific enough not to over-match unrelated output, an appropriate **retry** decision based on whether the failure is genuinely intermittent, a description of the failure, which tests/legs it affects, and links to example occurrences.

- **Reopen recurring KBEs rather than duplicating them.** When a failure matches a KBE that was already closed, it is appropriate to reopen that issue (noting the new occurrence) rather than filing a new one — particularly for a recent closure. Pair this with timing awareness: if the KBE was closed by a fix and the failure came from a build that ran before (or right around when) the fix landed, that is expected and should be skipped; a failure that recurs well after the fix means the fix didn't hold and the issue should be reopened.

- **Optionally include an analysis / RCA** of the failure on the issue.

## Separation of concerns

This workflow is responsible for **detecting failures and opening the right kind of issue.** The one label it sets itself is the `Known Build Error` label, and **only** on the intermittent-failure issues it files with the KBE template — that label is what makes an issue a KBE. **Build-break (hard failure) issues must not get the `Known Build Error` label** (counterintuitive given the name, but that label is reserved for the intermittent/known-issue template).

**All other labeling — owner and area/triage labels — stays with the existing triage automation** — this workflow should not try to assign owners or area labels itself. The two working together is the goal: this workflow gets the issue filed with an RCA, and triage routes it to the right owner so they can work with Copilot to implement a fix or decide on a different direction.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.