Resolve remaining Vally tool-selection failures after prompt disambiguation
- Dominant language
- C#
- Stars
- 3.7k
- Forks
- 624
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 220
Description
## Summary
Follow-up to PR #3163.
Vally run `2026-08-12T00-16-01-224Z` improved from 94.7% to 97.6% after updating prompt wording and interaction requirements, but 19 tool-selection failures remain.
| Run | Passed | Failed | Pass rate |
|---|---:|---:|---:|
| `2026-08-11T22-37-32-525Z` | 765/808 | 43 | 94.7% |
| `2026-08-12T00-16-01-224Z` | 786/805 | 19 | 97.6% |
Failure breakdown:
- 12 selected the correct MCP family but used the wrong or missing canonical command.
- 7 selected the wrong MCP family.
- No trials failed because the agent made no tool call.
## Action items
### Resolve overlapping tool ownership
- [ ] App Service diagnostics vs App Lens: two prompts explicitly requesting Azure App Service diagnostics still invoke `applens_resource_diagnose`. Clarify command ownership or accept App Lens in the grader.
- [ ] Azure Backup soft delete: `azurebackup_vault_update` invokes `azurebackup_governance_soft-delete`. Decide which command is canonical and remove or document the overlap.
- [ ] Foundry Extensions vs Foundry: a prompt explicitly requesting Microsoft Foundry Extensions still invokes `azmcp-foundry`. Align exposed wrapper names with grader expectations.
- [ ] Azure Files recommendations: both `fileshares_rec` prompts invoke `fileshares_fileshare_get`. Distinguish provisioning recommendations from resource details.
- [ ] SRE sub-agent deletion: `sreagent_agents_delete` invokes `sreagent_skills_delete`. Clarify the distinction between sub-agents and skills.
- [ ] Event Hubs namespace get: the prompt invokes Device Registry. Make Event Hubs ownership explicit, consistent with the create/delete prompt fixes.
- [ ] App Service listing: `Show me the web apps...` invokes `group_resource_list` instead of `appservice_webapp_get`.
### Improve canonical command discovery
- [ ] Azure Backup: cross-region restore invokes `azurebackup_vault_get`; protected-item status invokes `azurebackup_backup_status`; disable MUA does not invoke `azurebackup_security_configure-mua`.
- [ ] Monitor instrumentation: the Monitor family is selected but the canonical command is missing for `monitor_instrumentation_send-brownfield-analysis`, `monitor_instrumentation_send-enhancement-select`, and `monitor_instrumentation_get-learning-resource`.
- [ ] SRE architecture planning: the SRE Agent wrapper is selected without invoking `sreagent_architecture_plan`.
- [ ] Compute power state: aliases such as `vm stop` and `vm power-state stop` are used instead of `compute_vm_power-state`.
### Address wrong-family routes
- [ ] Compute disk encryption invokes `extension_cli_generate` instead of `compute_disk_update`.
- [ ] Application Insights error-rate analysis wanders through workspace/log-query commands instead of `monitor_metrics_query`.
- [ ] The remaining generic onboarding learning-resource prompt routes through SRE Agent and documentation instead of Azure Monitor instrumentation.
### Confirm reproducibility
Ten failures passed in the preceding run, including cross-region restore, MUA, protected-item status, disk encryption, VM stop, Event Hubs namespace get, and two Monitor submission prompts.
- [ ] Run each remaining failure at least three times.
- [ ] Treat a failure as deterministic only when it reproduces consistently.
- [ ] Avoid additional prompt rewrites for non-reproducible failures.
## Acceptance criteria
- Every `none` prompt invokes its expected MCP family and canonical command.
- Overlapping commands have one documented canonical owner.
- Foundry wrapper names match grader expectations.
- Deterministic failures reproduce across at least three trials before further changes.
- Three consecutive runs achieve at least a 99% pass rate.
Contributor guide
Assessment
This issue has not been assessed yet.