[ci-scan-feedback] Guard KBE signatures against shared multi-test Helix console-log catch-alls
- Dominant language
- C#
- Stars
- 18.3k
- Forks
- 5.6k
- PR merge metrics
- PR metrics pending
Description
## Triggering signals
- **#131449 / tracker #128742** — noahfalk [MEMBER]: the profiler KBE matched the **shared multi-test Helix console log**, pairing a per-run `Running test:` banner with a generic profiler-harness error line, so **every** profiler test failure in that workitem bucketed onto the one KBE. He manually corrected the KBE and asked that generation avoid matching shared-console error text, or capture a single expression that is both error-indicating **and** test-specific. (https://github.com/dotnet/runtime/issues/131449, https://github.com/dotnet/runtime/issues/128742)
- Rubric finding: `create-kbe.instructions.md`'s existing PASS/SKIP hard gate (§7) catches signatures that match `[PASS]`/`[SKIP]` lines, but does **not** catch a signature that matches *sibling tests'* failures in a shared console log — a distinct catch-all failure mode.
## Proposed edits
- `.github/workflows/shared/create-kbe.instructions.md` (§7, after the PASS/SKIP negative-output gate) — add a **"Shared multi-test Helix console log (catch-all gate)"** rule: when the log is a Helix per-workitem console log running many tests, require the most-specific element to be unique to the *failing test* (not the *workitem*), with three remediations (extend the array with a failing-path-only line, set `ExcludeConsoleLog: true` + a leg/xunit-level signature, or skip with `shared-log catch-all risk, needs human review`). Rationale: directly addresses noahfalk's #131449 over-matching.
- `.github/workflows/shared/create-kbe.instructions.md` (Bad vs good signatures table) — add a Bad/Good row using the concrete #131449 profiler pattern (banner + harness message → catch-all) so the rule is illustrated by example.
## Expected behavior change
The next scanner run will no longer file a KBE whose signature, though it greps positive and avoids PASS/SKIP lines, is scraped from a shared Helix console log and pairs a per-run banner with a generic error message. Such signatures either gain a failing-test-specific anchor, switch to a leg-level signature via `ExcludeConsoleLog: true`, or are skipped for human review — preventing the catch-all buckets that combined unrelated profiler failures onto a single KBE.
> [!NOTE]
> This pull request was generated by an AI agent (GitHub Copilot).
> Generated by [CI Outer-Loop Failure Scanner — Feedback](https://github.com/dotnet/runtime/actions/runs/32574800673) · opus48 · 443.4 AIC · ⌖ 25.3 AIC · ⊞ 20.6K · [◷](https://github.com/search?q=repo%3Adotnet%2Fruntime+%22gh-aw-workflow-id%3A+ci-failure-scan-feedback%22&type=pullrequests)
---
> [!NOTE]
> This was originally intended as a pull request, but the git push operation failed.
>
> **Original error:** The process '/usr/bin/git' failed with exit code 1
>
> **Workflow Run:** [View run details and download bundle artifact](https://github.com/dotnet/runtime/actions/runs/32574800673)
>
> The bundle file is available in the `agent` artifact in the workflow run linked above.
To create a pull request with the changes:
```sh
# Download the artifact from the workflow run
gh run download 32574800673 -n agent -D /tmp/agent-32574800673
# Fetch the bundle into a temporary ref, then update the local branch
git fetch /tmp/agent-32574800673/aw-ci-scan-feedback-console-log-catchall-20260822.bundle refs/heads/ci-scan-feedback/console-log-catchall-20260822:refs/bundles/create-pr-ci-scan-feedback-console-log-catchall-20260822-93341558b4632209-22d6adf7
git update-ref refs/heads/ci-scan-feedback/console-log-catchall-20260822-93341558b4632209 refs/bundles/create-pr-ci-scan-feedback-console-log-catchall-20260822-93341558b4632209-22d6adf7
git checkout ci-scan-feedback/console-log-catchall-20260822-93341558b4632209
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d refs/bundles/create-pr-ci-scan-feedback-console-log-catchall-20260822-93341558b4632209-22d6adf7
# Push the branch to origin
git push origin ci-scan-feedback/console-log-catchall-20260822-93341558b4632209
# Create the pull request
gh pr create --title '[ci-scan-feedback] Guard KBE signatures against shared multi-test Helix console-log catch-alls' --base main --head ci-scan-feedback/console-log-catchall-20260822-93341558b4632209 --repo dotnet/runtime
```
Contributor guide
Research direction
Start with .github/workflows/shared/create-kbe.instructions.md, reading §7 and the Bad vs good signatures table alongside the referenced #131449 pattern. Update the guidance so shared multi-test Helix console logs require a failing-test-specific anchor or the documented alternatives, and include the concrete bad/good example. Done means the catch-all rule and its three remediations are unambiguous.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions
- Domain
- ci-cd, documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100