Azure / Azure/azure-sdk-tools

Investigate whether the azure-sdk-mcp: tool-name qualifier in skills aids tool discovery

Open
#16,093 0 comments 0 reactions 0 assignees View on GitHub
needs-triage
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
143

Description

## Summary

In our Agent Skills (`.github/skills/azsdk-common-*`), tool names are sometimes written with the MCP server qualifier prefix `azure-sdk-mcp:` (e.g. `` `azure-sdk-mcp:azsdk_run_generate_sdk` ``) and sometimes without (e.g. `` `azsdk_run_generate_sdk` ``). PR #16089 standardized a couple of references to the qualified form for internal consistency, which surfaced a broader question.

## Question

Does prefixing tool names with the `azure-sdk-mcp:` server qualifier in skill prose actually improve **tool discovery / routing** by the agent, or is it purely a stylistic convention we have introduced?

Raised by @samvaity in review of #16089:
> Unrelated to this PR but we should really check if adding the specific mcp qualifier helps with discovery or just a preference that we have introduced in the skills.

## Why it matters

- If the qualifier measurably helps the model pick the right tool, we should adopt it consistently across **all** skills (and document it in the skill-authoring guidance).
- If it makes no difference (the model matches on the bare `azsdk_*` tool name regardless), we can drop it to reduce noise/verbosity and avoid brittle coupling to the server name.

## Suggested investigation

1. Audit current usage: count qualified vs unqualified tool-name references across `.github/skills/`.
2. Run a small Vally eval A/B (qualified vs unqualified prose) on a few tool-routing stimuli to see if selection accuracy changes.
3. Decide on a single convention and document it in the `skill-authoring` skill / engsys skills guidance.
4. Apply the chosen convention consistently across skills.

## Notes

- The eval graders match on the tool-call **name** (`azsdk_*`), not the prose, so this is purely a discovery/authoring concern, not a grading one.
- Follow-up to #16072 / #16089.

Contributor guide

Open the contributing guide

Research direction

Start by auditing qualified and unqualified tool-name references under .github/skills/. Run the proposed Vally A/B eval on representative tool-routing stimuli, then review the skill-authoring or engsys guidance. Done means choosing one convention, documenting it, and applying it consistently across the skills.

Written by the indexing model from the issue text.

Assessment

Domain
ai-infra-agents, tooling
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.