[ci-scan-feedback] Prevent shared-Helix-log catch-all KBE signatures
- Dominant language
- C#
- Stars
- 18.3k
- Forks
- 5.6k
- PR merge metrics
- PR metrics pending
Description
## Triggering signals
- **KBE #131449** (`[ci-scan]`, open) — MEMBER **janvorli** (2026-08-07): *"This issue seems to be a kitchen sink for all kinds of profiler tests failures."* ([comment](https://github.com/dotnet/runtime/issues/131449#issuecomment-5217618550))
- **KBE #131449** — MEMBER **noahfalk** (2026-08-22): *"Previously the KBE matched the Helix console log which runs many tests. Whenever some other profiler test failed the full log would still contain the line saying the nongcheap.cmd test ran as well as the error message, just that the error was part of a different test's output."* ([comment](https://github.com/dotnet/runtime/issues/131449#issuecomment-5378208122))
- **Tracker #128742** — MEMBER **noahfalk** (2026-08-22): *"the KBE generation logic either ideally either needs to avoid matching error message in the console log, or we need to capture a single console output expression that both indicates an error and is test-specific."* ([comment](https://github.com/dotnet/runtime/issues/128742#issuecomment-5376694491))
Rubric finding: KBE #131449's array-form `ErrorMessage` paired `Running test: profiler/gc/nongcheap/nongcheap.cmd` (announcement line, present in the profiler suite's single shared Helix console log for every test) with `Profiler tests are expected to contain the text 'PROFILER TEST PASSES'` (a shared-harness error any profiler test can print). Both lines coexist in the same console log, so the signature matched whenever *any* `profiler/**` test failed — a catch-all.
## Proposed edits
- `.github/workflows/shared/create-kbe.instructions.md` (array-form "Rules" list, ~L445) — new rule: do not pair a per-run announcement line with a non-test-specific error line when a leg runs many tests through one shared Helix console log; anchor every element to the failing test's own output, or set `ExcludeConsoleLog: true`. Ties directly to noahfalk's console-log-matching feedback.
- `.github/workflows/shared/create-kbe.instructions.md` ("Bad vs good signatures" table, ~L556) — new Bad/Good row using the #131449 signature as the concrete example.
## Expected behavior change
The next scanner run will not emit array-form KBEs whose elements individually match a shared multi-test Helix console log. Signatures that can only be anchored to suite-wide output will either be tightened to the failing test's own lines or set `ExcludeConsoleLog: true`, avoiding the profiler-style catch-all KBE that buckets unrelated failures.
> [!NOTE]
> This PR body was generated by the `ci-failure-scan-feedback` agentic workflow (AI/Copilot-generated).
> Generated by [CI Outer-Loop Failure Scanner — Feedback](https://github.com/dotnet/runtime/actions/runs/33510401456) · opus48 · 259.5 AIC · ⌖ 25.4 AIC · ⊞ 20.6K · [◷](https://github.com/search?q=repo%3Adotnet%2Fruntime+%22gh-aw-workflow-id%3A+ci-failure-scan-feedback%22&type=pullrequests)
---
> [!NOTE]
> This was originally intended as a pull request, but the git push operation failed.
>
> **Original error:** The process '/usr/bin/git' failed with exit code 1
>
> **Workflow Run:** [View run details and download bundle artifact](https://github.com/dotnet/runtime/actions/runs/33510401456)
>
> The bundle file is available in the `agent` artifact in the workflow run linked above.
To create a pull request with the changes:
```sh
# Download the artifact from the workflow run
gh run download 33510401456 -n agent -D /tmp/agent-33510401456
# Fetch the bundle into a temporary ref, then update the local branch
git fetch /tmp/agent-33510401456/aw-ci-scan-feedback-kbe-shared-helix-log-catchall.bundle refs/heads/ci-scan-feedback/kbe-shared-helix-log-catchall:refs/bundles/create-pr-ci-scan-feedback-kbe-shared-helix-log-catchall-77d6bd95fa74a85b-75f93697
git update-ref refs/heads/ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b refs/bundles/create-pr-ci-scan-feedback-kbe-shared-helix-log-catchall-77d6bd95fa74a85b-75f93697
git checkout ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d refs/bundles/create-pr-ci-scan-feedback-kbe-shared-helix-log-catchall-77d6bd95fa74a85b-75f93697
# Push the branch to origin
git push origin ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b
# Create the pull request
gh pr create --title '[ci-scan-feedback] Prevent shared-Helix-log catch-all KBE signatures' --base main --head ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b --repo dotnet/runtime
```
Contributor guide
Research direction
Read .github/workflows/shared/create-kbe.instructions.md, especially the Rules list around L445 and the Bad vs good signatures table around L556. Compare the proposed guidance with KBE #131449 and tracker #128742; it is done when the instructions cover shared Helix-log catch-all signatures and explain how to anchor or exclude suite-wide output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions
- Domain
- ci-cd, documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100