Align CI fixer and feedback evals with GitHub MCP policies
- Dominant language
- C#
- Stars
- 18.3k
- Forks
- 5.6k
- PR merge metrics
- PR metrics pending
Description
## Problem
The production `ci-failure-fix` and `ci-failure-scan-feedback` workflows require issue, pull request, and comment content to be read through the integrity-gated GitHub MCP tools. Their Vally eval stimuli and tool-call graders still instruct and reward shell `gh` reads instead.
This means `/ci-fix eval` and `/ci-feedback eval` exercise a different retrieval path than production and can pass without validating the workflows' MCP-only content-read policies. This mismatch surfaced while updating the shared MCP authentication plumbing in #132619; that PR intentionally aligns only the `ci-failure-scan` eval.
## Proposed work
- Update the fixer eval stimulus to discover KBEs with scoped `search_issues` calls and read issue bodies/comments with `issue_read`.
- Update the feedback eval stimulus to use scoped `search_issues` and `search_pull_requests`, then read issue/PR bodies and comments through `issue_read` and `pull_request_read`.
- Retain shell `gh` only for workflow metadata and repository-owned artifact enumeration explicitly permitted by each production prompt.
- Update tool-call graders to require the canonical `github-*` MCP calls and reject shell-based GitHub content reads.
- Run the Docker-backed live evals to confirm both workflows satisfy the updated contracts.
- Update `.github/workflows/evals/README.md` to describe the MCP-backed behavior accurately.
> [!NOTE]
> This issue was created with GitHub Copilot.
Contributor guide
Research direction
Start with the production ci-failure-fix and ci-failure-scan-feedback workflows, their Vally eval stimuli, and the tool-call graders; compare them with the MCP-aligned ci-failure-scan eval and the shared authentication changes from #132619. Run the Docker-backed live evals and update .github/workflows/evals/README.md. Done means content reads require the canonical github-* MCP calls, shell gh is limited to permitted metadata and artifact enumeration, and both evals pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github, github-actions, shell
- Domain
- ci-cd, documentation, testing-qa, tooling
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100