microsoft / microsoft/mcp

Tool Evaluations: Add CI pipeline support for evaluation experiments

Open
#3,038 1 comment 0 reactions 1 assignee Claimed by @LarryOsterman View on GitHub
enhancement server-Azure.Mcp tools-ToolSelection
Dominant language
C#
Stars
3.7k
Forks
624
Avg merge
2d 20h
Merged PRs (30d)
220

Description

## Summary

Integrate the skill evaluation experiment runner into CI pipelines so that experiments are run automatically on PRs and nightly.

**Depends on:** #3037

## Requirements

### PR Pipeline

- When a PR includes changes to experiments, evaluations, or tools, automatically run the affected experiments
- Detect changes in:
- `eng/tools/ToolDescriptionEvaluator/` (evaluator code or prompt files)
- `servers/Azure.Mcp.Server/docs/e2eTestPrompts.md` (source prompts)
- `core/Azure.Mcp.Core/src/Areas/Server/Resources/consolidated-tools.json` (consolidated tool definitions)
- Tool source code (`tools/Azure.Mcp.Tools.*/`) that would affect tool descriptions
- Run only the relevant variant experiments (namespace and/or consolidated) for changed areas
- Report results as PR comments or status checks with pass/fail based on configurable thresholds
- Upload JSON experiment results as pipeline artifacts

### Nightly Pipeline

- Run the full experiment suite across all tools and both variants (namespace, consolidated)
- Each run produces JSON result files that feed into the dashboards (#3037)
- Store historical results for trend analysis
- Alert on significant regressions (e.g., top choice success rate drops below a configured threshold)

### Pipeline Configuration

- Experiments should run in `--ci` mode with graceful handling of missing credentials
- Azure OpenAI credentials for embeddings should be sourced from pipeline secrets/variables
- Pipeline should regenerate grouped prompt files from `e2eTestPrompts.md` before running (no stale data)

### Acceptance Criteria

- [ ] PR pipeline runs experiments for changed experiments/evaluations/tools
- [ ] PR pipeline reports results as status checks or comments
- [ ] Nightly pipeline runs full experiment suite across all tools and both variants
- [ ] JSON results are uploaded as pipeline artifacts
- [ ] Nightly results feed into dashboards (#3037)
- [ ] Regression alerting is configured with adjustable thresholds
- [ ] Pipeline uses `--ci` mode and handles credentials from pipeline secrets

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.