Tool Evaluations: Add CI pipeline support for evaluation experiments
- Dominant language
- C#
- Stars
- 3.7k
- Forks
- 624
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 220
Description
## Summary
Integrate the skill evaluation experiment runner into CI pipelines so that experiments are run automatically on PRs and nightly.
**Depends on:** #3037
## Requirements
### PR Pipeline
- When a PR includes changes to experiments, evaluations, or tools, automatically run the affected experiments
- Detect changes in:
- `eng/tools/ToolDescriptionEvaluator/` (evaluator code or prompt files)
- `servers/Azure.Mcp.Server/docs/e2eTestPrompts.md` (source prompts)
- `core/Azure.Mcp.Core/src/Areas/Server/Resources/consolidated-tools.json` (consolidated tool definitions)
- Tool source code (`tools/Azure.Mcp.Tools.*/`) that would affect tool descriptions
- Run only the relevant variant experiments (namespace and/or consolidated) for changed areas
- Report results as PR comments or status checks with pass/fail based on configurable thresholds
- Upload JSON experiment results as pipeline artifacts
### Nightly Pipeline
- Run the full experiment suite across all tools and both variants (namespace, consolidated)
- Each run produces JSON result files that feed into the dashboards (#3037)
- Store historical results for trend analysis
- Alert on significant regressions (e.g., top choice success rate drops below a configured threshold)
### Pipeline Configuration
- Experiments should run in `--ci` mode with graceful handling of missing credentials
- Azure OpenAI credentials for embeddings should be sourced from pipeline secrets/variables
- Pipeline should regenerate grouped prompt files from `e2eTestPrompts.md` before running (no stale data)
### Acceptance Criteria
- [ ] PR pipeline runs experiments for changed experiments/evaluations/tools
- [ ] PR pipeline reports results as status checks or comments
- [ ] Nightly pipeline runs full experiment suite across all tools and both variants
- [ ] JSON results are uploaded as pipeline artifacts
- [ ] Nightly results feed into dashboards (#3037)
- [ ] Regression alerting is configured with adjustable thresholds
- [ ] Pipeline uses `--ci` mode and handles credentials from pipeline secrets
Contributor guide
Assessment
This issue has not been assessed yet.