Azure / Azure/azure-sdk-tools

Extend PromptToToolMatchEvaluator for CLI commands

Open
#13,759 0 comments 0 reactions 0 assignees View on GitHub
AzSDK Tools Agent azsdk-cli
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
143

Description

## Background
Following the work in #13710 (PromptToToolMatchEvaluator for MCP tools), we should extend the evaluation framework to also cover CLI commands.

### Why CLI needs separate evaluation
MCP tools and CLI commands have **separate descriptions** that can drift apart:

```csharp
// MCP Tool (via McpServerTool attribute)
[McpServerTool(Name = "azsdk_package_run_check"), Description("Run validation checks...")]

// CLI Command (via McpCommand in GetCommand())
new McpCommand("check", "Runs checks for a specified package...", ToolName)
```

As Claude/Copilot CLI and Codex usage increases, we want to ensure CLI command descriptions are also discoverable from natural language prompts.

## Proposed Work

### 1. CLI Command Discovery
- Create `CliCommandRegistry` that uses reflection to find all `McpCommand` instances
- Extract command name, description, and hierarchy (e.g., `azsdk package check`)
- More complex than MCP's `ListToolsAsync()` since it requires reflection over the command tree

### 2. Test Data
- New file: `TestPrompts.CLI.json` with CLI-specific prompts
- Different prompt style may be needed for CLI scenarios

### 3. Evaluator
- Reuse existing `EvaluationHelper.GenerateEmbeddingsAsync()` and `CalculateCosineSimilarity()`
- Either create `CliPromptToCommandMatchEvaluator` or parameterize existing evaluator

## Acceptance Criteria
- [ ] CLI commands can be discovered programmatically
- [ ] Test prompts defined for CLI commands
- [ ] Embedding-based evaluation runs against CLI command descriptions
- [ ] Coverage test ensures all CLI commands have test prompts

## References
- Parent PR: #13710
- Discussion: CLI and MCP tool descriptions don't always match 1:1

/cc @benbp

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the PromptToToolMatchEvaluator work in #13710 and the existing EvaluationHelper.GenerateEmbeddingsAsync() and CalculateCosineSimilarity() methods. Investigate how McpCommand instances and their command tree can be discovered, then define TestPrompts.CLI.json and choose whether to add a separate evaluator or parameterize the existing one. Done means CLI discovery, prompt coverage, embedding evaluation, and a coverage test for untested commands.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
cli, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.