dotnet / dotnet/runtime

Align CI fixer and feedback evals with GitHub MCP policies

Open
#132,634 1 comment 0 reactions 0 assignees View on GitHub
agentic-workflows area-Infrastructure untriaged
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

## Problem

The production `ci-failure-fix` and `ci-failure-scan-feedback` workflows require issue, pull request, and comment content to be read through the integrity-gated GitHub MCP tools. Their Vally eval stimuli and tool-call graders still instruct and reward shell `gh` reads instead.

This means `/ci-fix eval` and `/ci-feedback eval` exercise a different retrieval path than production and can pass without validating the workflows' MCP-only content-read policies. This mismatch surfaced while updating the shared MCP authentication plumbing in #132619; that PR intentionally aligns only the `ci-failure-scan` eval.

## Proposed work

- Update the fixer eval stimulus to discover KBEs with scoped `search_issues` calls and read issue bodies/comments with `issue_read`.
- Update the feedback eval stimulus to use scoped `search_issues` and `search_pull_requests`, then read issue/PR bodies and comments through `issue_read` and `pull_request_read`.
- Retain shell `gh` only for workflow metadata and repository-owned artifact enumeration explicitly permitted by each production prompt.
- Update tool-call graders to require the canonical `github-*` MCP calls and reject shell-based GitHub content reads.
- Run the Docker-backed live evals to confirm both workflows satisfy the updated contracts.
- Update `.github/workflows/evals/README.md` to describe the MCP-backed behavior accurately.

> [!NOTE]
> This issue was created with GitHub Copilot.

Contributor guide

Open the contributing guide

Research direction

Start with the production ci-failure-fix and ci-failure-scan-feedback workflows, their Vally eval stimuli, and the tool-call graders; compare them with the MCP-aligned ci-failure-scan eval and the shared authentication changes from #132619. Run the Docker-backed live evals and update .github/workflows/evals/README.md. Done means content reads require the canonical github-* MCP calls, shell gh is limited to permitted metadata and artifact enumeration, and both evals pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github, github-actions, shell
Domain
ci-cd, documentation, testing-qa, tooling
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.