microsoft / microsoft/mcp

Tool Evaluations: Flesh out existing evaluation experiments to include more tool coverage

Open
#3,036 0 comments 0 reactions 1 assignee Claimed by @LarryOsterman View on GitHub
enhancement server-Azure.Mcp tools-ToolSelection
Dominant language
C#
Stars
3.7k
Forks
624
Avg merge
2d 20h
Merged PRs (30d)
220

Description

## Summary

Expand evaluation prompt coverage to include all Azure MCP server tools across all three variants (baseline, namespace, consolidated), and ensure grouped prompt files stay in sync with the source of truth.

## Background

The e2e test prompts file (`servers/Azure.Mcp.Server/docs/e2eTestPrompts.md`) currently contains ~62 service areas with tool-specific prompts in markdown table format. The `Generate-GroupedPromptsJson.ps1` script transforms these into namespace-grouped and consolidated-grouped JSON files. However:

- Not all tools have corresponding evaluation prompts
- The pipeline requires manual regeneration of grouped prompt files
- Coverage gaps mean some tools are unevaluated in namespace and/or consolidated variants

## Requirements

### Expand Prompt Coverage

- Audit all tools exposed by the Azure MCP server and identify those missing from `e2eTestPrompts.md`
- Add evaluation prompts for every tool, ensuring each has at least 2-3 prompts that exercise realistic user queries
- Ensure prompts correctly map to tools in all three modes:
- **Baseline** (`prompts.json`): per-tool prompts
- **Namespace** (`namespace-prompts.json`): prompts grouped by service namespace
- **Consolidated** (`consolidated-prompts.json`): prompts grouped by consolidated tool mapping

### Automated Prompt Synchronization

- Update the experiment runner workflow to:
1. Run `Generate-PromptsJsonFromMarkdown.ps1` to regenerate `prompts.json` from `e2eTestPrompts.md`
2. Run `Generate-GroupedPromptsJson.ps1 -Mode Both` to regenerate `namespace-prompts.json` and `consolidated-prompts.json`
3. Then execute the evaluator

- Add a `--skip-regenerate` flag to bypass prompt regeneration for iterative development.

### Validation

- If `e2eTestPrompts.md` has been modified more recently than the generated JSON files, emit a warning suggesting regeneration
- The experiment runner should fail fast if any required prompt file is missing rather than silently running with stale data

### Acceptance Criteria

- [ ] Every tool in the Azure MCP server has evaluation prompts in `e2eTestPrompts.md`
- [ ] `namespace-prompts.json` and `consolidated-prompts.json` are regenerated before each experiment run by default
- [ ] A `--skip-regenerate` flag is available for iterative development
- [ ] Staleness warnings are emitted when JSON files are older than the source markdown
- [ ] Missing prompt files produce a clear error message
- [ ] Coverage report shows 100% of tools have at least one evaluation prompt

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.