Azure / Azure/azure-sdk-tools

Investigate how we can effectively develop auto evals for Github Copilot agent projects

Open
#11,320 0 comments 0 reactions 0 assignees View on GitHub
Evals
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
144

Description

We have AI projects that use this GHC agent with copilot instructions + MCP to do some task. It's hard to evaluate how changes to instructions or MCP tools may affect the overall experience (calling the wrong tool, missing something in the sequence of steps, getting off task, etc). There isn't a GHC agent API AFAIK, so this makes evaluating a change more of a manual process. We need to figure out a way to automate evaluating the agent so we can quickly understand impact of changes.

Some brainstorm:
- Export the full conversation transcript and run auto evals on that
- Use playwright or MCP sampling to drive conversation?

Contributor guide

Open the contributing guide

Research direction

Start by comparing the proposed full-transcript, Playwright, and MCP sampling approaches for evaluating GitHub Copilot agent projects. Define representative cases for wrong tool calls, incomplete sequences, and off-task behavior; done should be a documented, automatable evaluation approach with a way to measure changes in agent experience.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, playwright
Domain
ai, developer-experience, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.