move azure-mcp evals script and hook into live CI
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 143
Description
Work started here (https://github.com/Azure/azure-sdk-tools/tree/main/tools/ai-evals/azure-mcp) and now we want to move it to azure-mcp repo
This issue tracks hooking the evals script up to pipeline using live resources: https://github.com/Azure/azure-mcp/blob/main/eng/pipelines/templates/jobs/live-test.yml
Additional work: Investigate how we can apply evals to do A/B testing of prompts and measure prompt token usage ([#40](https://github.com/Azure/azure-mcp-pr/issues/40))
Contributor guide
Research direction
Start in tools/ai-evals/azure-mcp and inspect eng/pipelines/templates/jobs/live-test.yml in the azure-mcp repository. Trace how the existing evals script uses live resources and how the pipeline template invokes live tests. Done means the evals work is moved to azure-mcp and runs through live CI; the A/B prompt and token-usage investigation is tracked separately in issue #40.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, github-actions
- Domain
- ci-cd, cloud, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100