Extend PromptToToolMatchEvaluator for CLI commands
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 143
Description
## Background
Following the work in #13710 (PromptToToolMatchEvaluator for MCP tools), we should extend the evaluation framework to also cover CLI commands.
### Why CLI needs separate evaluation
MCP tools and CLI commands have **separate descriptions** that can drift apart:
```csharp
// MCP Tool (via McpServerTool attribute)
[McpServerTool(Name = "azsdk_package_run_check"), Description("Run validation checks...")]
// CLI Command (via McpCommand in GetCommand())
new McpCommand("check", "Runs checks for a specified package...", ToolName)
```
As Claude/Copilot CLI and Codex usage increases, we want to ensure CLI command descriptions are also discoverable from natural language prompts.
## Proposed Work
### 1. CLI Command Discovery
- Create `CliCommandRegistry` that uses reflection to find all `McpCommand` instances
- Extract command name, description, and hierarchy (e.g., `azsdk package check`)
- More complex than MCP's `ListToolsAsync()` since it requires reflection over the command tree
### 2. Test Data
- New file: `TestPrompts.CLI.json` with CLI-specific prompts
- Different prompt style may be needed for CLI scenarios
### 3. Evaluator
- Reuse existing `EvaluationHelper.GenerateEmbeddingsAsync()` and `CalculateCosineSimilarity()`
- Either create `CliPromptToCommandMatchEvaluator` or parameterize existing evaluator
## Acceptance Criteria
- [ ] CLI commands can be discovered programmatically
- [ ] Test prompts defined for CLI commands
- [ ] Embedding-based evaluation runs against CLI command descriptions
- [ ] Coverage test ensures all CLI commands have test prompts
## References
- Parent PR: #13710
- Discussion: CLI and MCP tool descriptions don't always match 1:1
/cc @benbp
Contributor guide
Research direction
Start by reviewing the PromptToToolMatchEvaluator work in #13710 and the existing EvaluationHelper.GenerateEmbeddingsAsync() and CalculateCosineSimilarity() methods. Investigate how McpCommand instances and their command tree can be discovered, then define TestPrompts.CLI.json and choose whether to add a separate evaluator or parameterize the existing one. Done means CLI discovery, prompt coverage, embedding evaluation, and a coverage test for untested commands.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- cli, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100