microsoft / microsoft/mcp

Tool Evaluations: Create task-based holistic prompts for variant comparison

Open
#3,039 0 comments 0 reactions 1 assignee Claimed by @LarryOsterman View on GitHub
enhancement server-Azure.Mcp tools-ToolSelection
Dominant language
C#
Stars
3.7k
Forks
624
Avg merge
2d 20h
Merged PRs (30d)
220

Description

## Summary

Create a new set of **task-based prompts** that exercise the Baseline/Namespace/Consolidated variants more realistically. These prompts describe user goals and workflows rather than targeting specific tools, simulating how real users interact with an AI assistant backed by Azure MCP tools.

**Depends on:** #3038

## Motivation

The existing e2e prompts in `e2eTestPrompts.md` validate that a specific tool is selected for a specific query (e.g., ""List all storage accounts in my subscription"" -> `storage_account_get`). This is valuable for per-tool regression testing but doesn't capture how the variant modes differ in practice:

- **Namespace mode** groups tools by service — a task-based prompt that spans multiple services tests whether the right namespace is selected first.
- **Consolidated mode** uses a single tool — task-based prompts test whether the consolidated router can disambiguate across all Azure services.
- **Baseline** (no tools) — task-based prompts establish the floor for how well prompts match without any MCP server.

Task-based prompts better reflect real user behavior and expose strengths/weaknesses of each variant's tool organization strategy.

## Requirements

### Prompt Design Principles

1. **Goal-oriented**: Describe what the user wants to accomplish, not which tool to use.
- Yes: ""I need to set up a new web application with a database and ensure it's secured properly""
- No: ""List all App Services in my subscription""

2. **Multi-service**: Many prompts should naturally span multiple Azure services.
- ""Help me debug why my Function App can't connect to my Cosmos DB""
- ""Set up monitoring and alerts for my AKS cluster""

3. **Scenario categories** (suggested, not exhaustive):
- **Provisioning**: ""Deploy a new storage account with a container for my application logs""
- **Troubleshooting**: ""My web app is returning 500 errors, help me investigate""
- **Security & compliance**: ""Audit the Key Vault access policies for my production subscription""
- **Cost optimization**: ""Which of my resources are costing the most and what can I do about it?""
- **Migration**: ""I need to move my SQL database to a different region""
- **Monitoring**: ""Set up alerts for when my Redis cache memory usage exceeds 80%""

4. **Varying complexity**: Include simple single-service tasks alongside complex multi-service workflows.

### File Organization

- Create `eng/tools/ToolDescriptionEvaluator/prompts/task-prompts.json` with task-based prompts.
- Each prompt should specify the **expected tool(s)** that should be selected, acknowledging that task-based prompts may validly map to multiple tools.
- Update `Generate-GroupedPromptsJson.ps1` to also produce `task-namespace-prompts.json` and `task-consolidated-prompts.json` variants if needed, or design the task prompts to work across all three variant modes directly.

### Expected Tool Mapping

Since task-based prompts may not map 1:1 to a single tool, the evaluation format should support:
- **Primary expected tools**: The tool(s) most likely to be selected first.
- **Acceptable tools**: Additional tools that would be valid selections for the task.
- Scoring should credit any match within the acceptable set.

### Suggested Initial Prompt Set

Start with 30-50 task-based prompts covering at least 10 distinct Azure service areas, with a mix of:
- ~15 single-service task prompts
- ~15 multi-service task prompts
- ~10 ambiguous/exploratory prompts (e.g., ""What's going on with my Azure resources?"")

### Acceptance Criteria

- [ ] `task-prompts.json` created with 30+ task-based prompts
- [ ] Prompts cover at least 10 Azure service areas
- [ ] Multi-tool expected matching is supported in the evaluation schema
- [ ] Experiment runner can use `--prompts-file task-prompts.json` to run task-based evaluations
- [ ] JSON summary output includes task-based results alongside per-tool results when both are run
- [ ] Results clearly show how each variant (baseline/namespace/consolidated) handles goal-oriented queries differently

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.