Tool Evaluations: Flesh out existing evaluation experiments to include more tool coverage
- Dominant language
- C#
- Stars
- 3.7k
- Forks
- 624
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 220
Description
## Summary
Expand evaluation prompt coverage to include all Azure MCP server tools across all three variants (baseline, namespace, consolidated), and ensure grouped prompt files stay in sync with the source of truth.
## Background
The e2e test prompts file (`servers/Azure.Mcp.Server/docs/e2eTestPrompts.md`) currently contains ~62 service areas with tool-specific prompts in markdown table format. The `Generate-GroupedPromptsJson.ps1` script transforms these into namespace-grouped and consolidated-grouped JSON files. However:
- Not all tools have corresponding evaluation prompts
- The pipeline requires manual regeneration of grouped prompt files
- Coverage gaps mean some tools are unevaluated in namespace and/or consolidated variants
## Requirements
### Expand Prompt Coverage
- Audit all tools exposed by the Azure MCP server and identify those missing from `e2eTestPrompts.md`
- Add evaluation prompts for every tool, ensuring each has at least 2-3 prompts that exercise realistic user queries
- Ensure prompts correctly map to tools in all three modes:
- **Baseline** (`prompts.json`): per-tool prompts
- **Namespace** (`namespace-prompts.json`): prompts grouped by service namespace
- **Consolidated** (`consolidated-prompts.json`): prompts grouped by consolidated tool mapping
### Automated Prompt Synchronization
- Update the experiment runner workflow to:
1. Run `Generate-PromptsJsonFromMarkdown.ps1` to regenerate `prompts.json` from `e2eTestPrompts.md`
2. Run `Generate-GroupedPromptsJson.ps1 -Mode Both` to regenerate `namespace-prompts.json` and `consolidated-prompts.json`
3. Then execute the evaluator
- Add a `--skip-regenerate` flag to bypass prompt regeneration for iterative development.
### Validation
- If `e2eTestPrompts.md` has been modified more recently than the generated JSON files, emit a warning suggesting regeneration
- The experiment runner should fail fast if any required prompt file is missing rather than silently running with stale data
### Acceptance Criteria
- [ ] Every tool in the Azure MCP server has evaluation prompts in `e2eTestPrompts.md`
- [ ] `namespace-prompts.json` and `consolidated-prompts.json` are regenerated before each experiment run by default
- [ ] A `--skip-regenerate` flag is available for iterative development
- [ ] Staleness warnings are emitted when JSON files are older than the source markdown
- [ ] Missing prompt files produce a clear error message
- [ ] Coverage report shows 100% of tools have at least one evaluation prompt
Contributor guide
Assessment
This issue has not been assessed yet.