[EPIC] Experiment-Driven Tool Evaluation

@LarryOsterman is already working on this.

Since Jul 14, 2026.

## Problem statement

Azure MCP exposes baseline, namespace, and consolidated tool surfaces whose quality cannot be judged from implementation alone. Tool descriptions, routing, prompt coverage, confidence, and end-to-end task success need reproducible experiments so product decisions are based on measured agent behavior rather than isolated examples.

## Vision

Every supported Azure MCP tool mode is evaluated through repeatable experiments with realistic tasks, structured results, dashboards, and CI signals that identify regressions and guide product decisions.

## Current work

- [ ] #3036, expand experiments across more tools and service areas
- [ ] #3037, report experiment results through dashboards
- [ ] #3038, add CI pipeline support for evaluation experiments
- [ ] #3039, create task-based holistic prompts for variant comparison
- [ ] #2848, integrate Vally into CI

Related enforcement work such as #2497 is consumed through the PR quality guardrails epic.

## Completed foundation

- #238 evaluated Microsoft.Extensions.AI.Evaluation
- #1506 centralized ToolDescriptionEvaluator utilities
- #471 added managed-identity support to the evaluator
- The existing runner and prompt sets provide the baseline for namespace and consolidated comparisons

## Goals (in scope)

- Compare baseline, namespace, and consolidated modes using consistent datasets and metrics
- Expand prompt and tool coverage across representative Azure scenarios
- Add realistic multi-step and task-level evaluations beyond isolated tool matching
- Emit structured experiment results suitable for CI and dashboards
- Define regression thresholds and ownership for failed evaluations
- Use results to guide tool descriptions, routing, and mode decisions

## Non-goals (out of scope)

- Treating embedding similarity as the only product-quality signal
- Owning implementation of routing or protocol behavior being measured
- Blocking every pull request on experimental or statistically insignificant variation

## Success criteria

- [ ] Variant experiments run reproducibly against a documented baseline
- [ ] Representative service areas and holistic tasks have maintained datasets
- [ ] CI and dashboards expose comparable quality and confidence trends
- [ ] Material regressions produce actionable failures with responsible owners
- [ ] Product mode and routing decisions cite measured experiment results

## Dependencies

- Stable server modes and generated tool definitions
- Evaluation models, credentials, datasets, and Vally integration
- CI capacity and durable result storage
- PR guardrails that consume established evaluation thresholds