dotnet / dotnet/runtime

[ci-scan-feedback] Guard KBE signatures against shared multi-test Helix console-log catch-alls

Open Beginner friendly
#132,657 1 comment 0 reactions 0 assignees View on GitHub
agentic-workflows area-Infrastructure untriaged
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

## Triggering signals

- **#131449 / tracker #128742** — noahfalk [MEMBER]: the profiler KBE matched the **shared multi-test Helix console log**, pairing a per-run `Running test:` banner with a generic profiler-harness error line, so **every** profiler test failure in that workitem bucketed onto the one KBE. He manually corrected the KBE and asked that generation avoid matching shared-console error text, or capture a single expression that is both error-indicating **and** test-specific. (https://github.com/dotnet/runtime/issues/131449, https://github.com/dotnet/runtime/issues/128742)
- Rubric finding: `create-kbe.instructions.md`'s existing PASS/SKIP hard gate (§7) catches signatures that match `[PASS]`/`[SKIP]` lines, but does **not** catch a signature that matches *sibling tests'* failures in a shared console log — a distinct catch-all failure mode.

## Proposed edits

- `.github/workflows/shared/create-kbe.instructions.md` (§7, after the PASS/SKIP negative-output gate) — add a **"Shared multi-test Helix console log (catch-all gate)"** rule: when the log is a Helix per-workitem console log running many tests, require the most-specific element to be unique to the *failing test* (not the *workitem*), with three remediations (extend the array with a failing-path-only line, set `ExcludeConsoleLog: true` + a leg/xunit-level signature, or skip with `shared-log catch-all risk, needs human review`). Rationale: directly addresses noahfalk's #131449 over-matching.
- `.github/workflows/shared/create-kbe.instructions.md` (Bad vs good signatures table) — add a Bad/Good row using the concrete #131449 profiler pattern (banner + harness message → catch-all) so the rule is illustrated by example.

## Expected behavior change

The next scanner run will no longer file a KBE whose signature, though it greps positive and avoids PASS/SKIP lines, is scraped from a shared Helix console log and pairs a per-run banner with a generic error message. Such signatures either gain a failing-test-specific anchor, switch to a leg-level signature via `ExcludeConsoleLog: true`, or are skipped for human review — preventing the catch-all buckets that combined unrelated profiler failures onto a single KBE.

> [!NOTE]
> This pull request was generated by an AI agent (GitHub Copilot).

> Generated by [CI Outer-Loop Failure Scanner — Feedback](https://github.com/dotnet/runtime/actions/runs/32574800673) · opus48 · 443.4 AIC · ⌖ 25.3 AIC · ⊞ 20.6K · [◷](https://github.com/search?q=repo%3Adotnet%2Fruntime+%22gh-aw-workflow-id%3A+ci-failure-scan-feedback%22&type=pullrequests)

---

> [!NOTE]
> This was originally intended as a pull request, but the git push operation failed.
>
> **Original error:** The process '/usr/bin/git' failed with exit code 1
>
> **Workflow Run:** [View run details and download bundle artifact](https://github.com/dotnet/runtime/actions/runs/32574800673)
>
> The bundle file is available in the `agent` artifact in the workflow run linked above.

To create a pull request with the changes:

```sh
# Download the artifact from the workflow run
gh run download 32574800673 -n agent -D /tmp/agent-32574800673

# Fetch the bundle into a temporary ref, then update the local branch
git fetch /tmp/agent-32574800673/aw-ci-scan-feedback-console-log-catchall-20260822.bundle refs/heads/ci-scan-feedback/console-log-catchall-20260822:refs/bundles/create-pr-ci-scan-feedback-console-log-catchall-20260822-93341558b4632209-22d6adf7
git update-ref refs/heads/ci-scan-feedback/console-log-catchall-20260822-93341558b4632209 refs/bundles/create-pr-ci-scan-feedback-console-log-catchall-20260822-93341558b4632209-22d6adf7
git checkout ci-scan-feedback/console-log-catchall-20260822-93341558b4632209
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d refs/bundles/create-pr-ci-scan-feedback-console-log-catchall-20260822-93341558b4632209-22d6adf7

# Push the branch to origin
git push origin ci-scan-feedback/console-log-catchall-20260822-93341558b4632209

# Create the pull request
gh pr create --title '[ci-scan-feedback] Guard KBE signatures against shared multi-test Helix console-log catch-alls' --base main --head ci-scan-feedback/console-log-catchall-20260822-93341558b4632209 --repo dotnet/runtime
```

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/shared/create-kbe.instructions.md, reading §7 and the Bad vs good signatures table alongside the referenced #131449 pattern. Update the guidance so shared multi-test Helix console logs require a failing-test-specific anchor or the documented alternatives, and include the concrete bad/good example. Done means the catch-all rule and its three remediations are unambiguous.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.