Investigate how we can effectively develop auto evals for Github Copilot agent projects
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 144
Description
We have AI projects that use this GHC agent with copilot instructions + MCP to do some task. It's hard to evaluate how changes to instructions or MCP tools may affect the overall experience (calling the wrong tool, missing something in the sequence of steps, getting off task, etc). There isn't a GHC agent API AFAIK, so this makes evaluating a change more of a manual process. We need to figure out a way to automate evaluating the agent so we can quickly understand impact of changes.
Some brainstorm:
- Export the full conversation transcript and run auto evals on that
- Use playwright or MCP sampling to drive conversation?
Contributor guide
Research direction
Start by comparing the proposed full-transcript, Playwright, and MCP sampling approaches for evaluating GitHub Copilot agent projects. Define representative cases for wrong tool calls, incomplete sequences, and off-task behavior; done should be a documented, automatable evaluation approach with a way to measure changes in agent experience.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, playwright
- Domain
- ai, developer-experience, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100