microsoft / microsoft/GitHub-Copilot-for-Azure
Integration test failure: azure-quotas – region comparison CLI keyword missing when using MCP tool [Assertion mismatch]
- Dominant language
- Python
- Stars
- 250
- Forks
- 204
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 67
Description
## Prompt
```
Compare Azure compute quota availability across East US, West US 2, and Central US
```
## Summary
**Run:** [Integration Tests - all #132](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23848054525)
**Test:** `azure-quotas_ - Integration Tests › azure-quotas › handles region comparison query`
**Result:** Fail
**Duration:** 30.1s
## Root Cause Category
**Assertion mismatch**
## Diagnosis
### What was expected
The test asserts that the skill is invoked AND the response or tool call arguments include `"az quota"`:
```ts
// tests/azure-quotas/integration.test.ts:141
expect(isSkillUsed).toBe(true); // PASSED — skill was invoked
expect(mentionsQuotaCmd).toBe(true); // FAILED — "az quota" not found
```
Where:
````ts
const mentionsQuotaCmd = doesAssistantMessageIncludeKeyword(agentMetadata, "az quota")
|| doToolCallArgsIncludeKeyword(agentMetadata, "az quota");
```
### What actually happened
The agent correctly invoked the `azure-quotas` skill and successfully compared quota availability across all three regions (East US, West US 2, Central US). It used the MCP `azure-quota` tool (4 calls) — one to learn available commands, then one per region. The quota data was retrieved and a comprehensive comparison was provided.
However, because the agent exclusively used the MCP `azure-quota` tool rather than issuing `az quota` CLI commands, the string `"az quota"` did not appear in the assistant message or tool call arguments. The keyword assertion failed.
**Agent tool usage:**
```
tool: azure-quota → learn commands
tool: azure-quota → quota_usage_check eastus (100+ VM SKUs)
tool: azure-quota → quota_usage_check westus2
tool: azure-quota → quota_usage_check centralus
````
### Why it failed
Same root cause as the `provides quota check commands for compute resources` test (see #1628): the agent prefers the `azure-quota` MCP tool over the `az quota` CLI, and the test assertion does not account for the MCP execution path.
### Suggested fix
**Option A (fix the skill):** Update `azure-quotas` skill instructions to include `az quota` CLI commands in the response for transparency, even when MCP tool is used.
**Option B (fix the test):** Update the assertion to accept MCP tool invocations:
```ts
const mentionsQuotaCmd = doesAssistantMessageIncludeKeyword(agentMetadata, "az quota")
|| doToolCallArgsIncludeKeyword(agentMetadata, "az quota")
|| isToolCalled(agentMetadata, "azure-quota"); // accept MCP tool path
```
## azure-quotas Skill Invocation
| Skill | Invoked |
|-------|---------|
| **azure-quotas** | **Yes** |
## Skill Report Context
Per-test section from SKILL-REPORT.md (click to expand)
From artifact `integration-report-azure-quotas` (run #132):
- **Skill Invocation Success Rate:** 100% (10/10)
- **Average Confidence:** 97%
The agent retrieved complete quota data for all 3 regions, identified GPU/specialty compute requiring quota increases, and produced a regional comparison. The MCP tool (`azure-quota`) was used exclusively.
## Environment
- **Runner OS:** ubuntu-latest
- **Node.js:** v24.x
- **Model:** claude-sonnet-4.5
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23848054525
- **Commit:** `e948e2993efd30da197ff1b0d3e0c43092e1f3d5`
- **Test file:** `tests/azure-quotas/integration.test.ts:141`
Contributor guide
Assessment
This issue has not been assessed yet.