microsoft / microsoft/GitHub-Copilot-for-Azure
Integration test failure: azure-quotas – quota check CLI keyword missing when using MCP tool [Assertion mismatch]
- Dominant language
- Python
- Stars
- 250
- Forks
- 204
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 67
Description
## Prompt
```
Check my Azure VM quota limits and current usage in East US
```
## Summary
**Run:** [Integration Tests - all #132](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23848054525)
**Test:** `azure-quotas_ - Integration Tests › azure-quotas › provides quota check commands for compute resources`
**Result:** Fail
**Duration:** 24.9s
> **Note:** Existing issue #1450 tracks a related failure where the skill was **not invoked** at all. This is a different failure — the skill **was invoked**, but the response/tool-call args did not contain the `az quota` keyword because the agent used the MCP `azure-quota` tool instead of the Azure CLI.
## Root Cause Category
**Assertion mismatch**
## Diagnosis
### What was expected
The test asserts that the skill is invoked AND the response or tool call arguments include:
1. `"az quota"` (CLI command keyword) — in assistant message or tool call args
2. `"/subscriptions/"` (scope URI) — in assistant message or tool call args
````ts
// tests/azure-quotas/integration.test.ts:110
expect(isSkillUsed).toBe(true); // PASSED — skill was invoked
expect(mentionsQuotaCmd).toBe(true); // FAILED — "az quota" not found
expect(mentionsScope).toBe(true);
```
### What actually happened
The agent correctly invoked the `azure-quotas` skill and retrieved VM quota data for East US. However, it used the MCP `azure-quota` tool (not the Azure CLI `az quota` command):
```
tool: azure-quota
→ learn available commands
→ check VM quota usage in eastus
````
Since the agent used the MCP tool rather than issuing `az quota` CLI commands, the keyword `"az quota"` was absent from both the assistant message and tool call arguments. The assertion `mentionsQuotaCmd` evaluates to `false`.
### Why it failed
The `azure-quotas` skill (or agent behavior) prefers the `azure-quota` MCP tool over the `az quota` CLI when MCP is available. The test was written expecting the skill to produce `az quota` CLI commands, but the MCP path is a valid alternative. The test's keyword check doesn't cover MCP tool invocations.
### Suggested fix
**Option A (fix the skill):** Update `azure-quotas` skill instructions to always include `az quota` commands in the response alongside MCP tool usage.
**Option B (fix the test):** Update the assertion to also accept MCP tool invocations as a valid path:
```ts
const mentionsQuotaCmd = doesAssistantMessageIncludeKeyword(agentMetadata, "az quota")
|| doToolCallArgsIncludeKeyword(agentMetadata, "az quota")
|| isToolCalled(agentMetadata, "azure-quota"); // accept MCP tool
```
## azure-quotas Skill Invocation
| Skill | Invoked |
|-------|---------|
| **azure-quotas** | **Yes** |
## Skill Report Context
Per-test section from SKILL-REPORT.md (click to expand)
From artifact `integration-report-azure-quotas` (run #132):
- **Skill Invocation Success Rate:** 100% (10/10)
- **Overall Test Pass Rate:** 100% (14/14) ← per SKILL-REPORT (junit.xml shows 4/6 pass)
- **Average Confidence:** 97%
The skill was correctly invoked; quota data was successfully retrieved. The agent used MCP `azure-quota` tool exclusively, omitting `az quota` CLI syntax from the response.
## Environment
- **Runner OS:** ubuntu-latest
- **Node.js:** v24.x
- **Model:** claude-sonnet-4.5
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23848054525
- **Commit:** `e948e2993efd30da197ff1b0d3e0c43092e1f3d5`
- **Test file:** `tests/azure-quotas/integration.test.ts:110`
**Related:** #1450 (skill not invoked — a different root cause)
> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23854424383) · [◷](https://github.com/search?q=repo%3Amicrosoft%2FGitHub-Copilot-for-Azure+is%3Aissue+%22gh-aw-workflow-call-id%3A+microsoft%2FGitHub-Copilot-for-Azure%2Fanalyze-test-run%22&type=issues)
Contributor guide
Assessment
This issue has not been assessed yet.