microsoft / microsoft/GitHub-Copilot-for-Azure
Integration test failure: azure-quotas – VM quota compute scope assertion [Assertion mismatch]
- Dominant language
- Python
- Stars
- 250
- Forks
- 204
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 67
Description
## Prompt
```
Check my Azure VM quota limits and current usage in East US
```
## Summary
**Run:** [Integration Tests - all #169](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/24564577788)
**Test:** `azure-quotas_ - Integration Tests › azure-quotas > provides quota check commands for compute resources`
**Result:** Fail
**Duration:** 0m 32s
## Root Cause Category
**Assertion mismatch**
## Diagnosis
### What was expected
The test checks three assertions:
1. `isSkillInvoked(agentMetadata, "azure-quotas")` → `true`
2. `mentionsQuotaCmd` (assistant message or tool args include `"az quota"`) → `true`
3. `mentionsScope` (assistant message or tool args include `"/subscriptions/"`) → `true`
The failure is at line 110: `expect(mentionsScope).toBe(true)` — received `false`.
### What actually happened
The agent invoked the `azure-quota` MCP tool rather than issuing an `az quota` CLI command with an explicit `/subscriptions/{id}/providers/...` scope. The MCP tool abstracted away the subscription-scoped URI, so neither the assistant message nor the tool call arguments contained the `/subscriptions/` string pattern.
### Why it failed
The test assertion `mentionsScope` checks for the literal string `/subscriptions/` in tool call args or assistant output. When the skill uses the `azure-quota` MCP tool instead of a raw `az quota list --scope /subscriptions/...` CLI command, the subscription scope URI does not appear in the tracked output, causing the assertion to fail even though correct quota data was returned.
### Suggested fix
- Update the skill to include the `/subscriptions/{subscriptionId}/...` scope in the assistant response message (e.g. show the full scope used for the query)
- Or update the test assertion to also accept MCP-tool-based quota retrieval patterns (e.g., check for subscription ID in tool parameters)
- Alternatively, ensure the MCP tool call arguments include the subscription scope URI
## azure-quotas Skill Invocation
| Skill | Invoked |
|-------|---------|
| **azure-quotas** | **Yes** |
## Skill Report Context
Per-test section from SKILL-REPORT.md (click to expand)
**Test:** `azure-quotas_provides_quota_check_commands_for_compute_resources`
**Type:** Integration
**Prompt:** "Check my Azure VM quota limits and current usage in East US"
**Runs:** 1 | **Pass Rate:** 100% (per skill report) | **Avg Confidence:** 95%
**What Happened:**
The skill was invoked correctly after an initial discovery step. The agent used the `azure-quota` MCP tool to retrieve comprehensive VM quota data for East US. The tool returned quota information for all VM families (0–100 vCPUs). Response categorized VMs by type (General Purpose, Memory Optimized, Compute Optimized) with actionable recommendations.
**❌ What Went Wrong (per JUnit):**
The `mentionsScope` assertion failed — `/subscriptions/` was not present in assistant messages or tool call arguments because the azure-quota MCP tool abstracts the subscription scope.
## Environment
- **Runner OS:** ubuntu-latest
- **Node.js:** (from workflow)
- **Model:** claude-sonnet-4.5
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/24564577788
- **Commit:** `c4ea0f4e6395b6825085bbce7eb448dedf9deec5`
> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/24570398196/agentic_workflow) · ● 2.4M · [◷](https://github.com/search?q=repo%3Amicrosoft%2FGitHub-Copilot-for-Azure+is%3Aissue+%22gh-aw-workflow-call-id%3A+microsoft%2FGitHub-Copilot-for-Azure%2Fanalyze-test-run%22&type=issues)
Contributor guide
Assessment
This issue has not been assessed yet.