Investigate whether the azure-sdk-mcp: tool-name qualifier in skills aids tool discovery
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 143
Description
## Summary
In our Agent Skills (`.github/skills/azsdk-common-*`), tool names are sometimes written with the MCP server qualifier prefix `azure-sdk-mcp:` (e.g. `` `azure-sdk-mcp:azsdk_run_generate_sdk` ``) and sometimes without (e.g. `` `azsdk_run_generate_sdk` ``). PR #16089 standardized a couple of references to the qualified form for internal consistency, which surfaced a broader question.
## Question
Does prefixing tool names with the `azure-sdk-mcp:` server qualifier in skill prose actually improve **tool discovery / routing** by the agent, or is it purely a stylistic convention we have introduced?
Raised by @samvaity in review of #16089:
> Unrelated to this PR but we should really check if adding the specific mcp qualifier helps with discovery or just a preference that we have introduced in the skills.
## Why it matters
- If the qualifier measurably helps the model pick the right tool, we should adopt it consistently across **all** skills (and document it in the skill-authoring guidance).
- If it makes no difference (the model matches on the bare `azsdk_*` tool name regardless), we can drop it to reduce noise/verbosity and avoid brittle coupling to the server name.
## Suggested investigation
1. Audit current usage: count qualified vs unqualified tool-name references across `.github/skills/`.
2. Run a small Vally eval A/B (qualified vs unqualified prose) on a few tool-routing stimuli to see if selection accuracy changes.
3. Decide on a single convention and document it in the `skill-authoring` skill / engsys skills guidance.
4. Apply the chosen convention consistently across skills.
## Notes
- The eval graders match on the tool-call **name** (`azsdk_*`), not the prose, so this is purely a discovery/authoring concern, not a grading one.
- Follow-up to #16072 / #16089.
Contributor guide
Research direction
Start by auditing qualified and unqualified tool-name references under .github/skills/. Run the proposed Vally A/B eval on representative tool-routing stimuli, then review the skill-authoring or engsys guidance. Done means choosing one convention, documenting it, and applying it consistently across the skills.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai-infra-agents, tooling
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100