microsoft / microsoft/GitHub-Copilot-for-Azure

Integration test failure: azure-quotas – quota check CLI keyword missing when using MCP tool [Assertion mismatch]

Open
#1,628 0 comments 0 reactions 1 assignee Claimed by @rakal-dyh View on GitHub
azure-quotas bug integration-test test-failure
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Prompt

```
Check my Azure VM quota limits and current usage in East US
```

## Summary

**Run:** [Integration Tests - all #132](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23848054525)
**Test:** `azure-quotas_ - Integration Tests › azure-quotas › provides quota check commands for compute resources`
**Result:** Fail
**Duration:** 24.9s

> **Note:** Existing issue #1450 tracks a related failure where the skill was **not invoked** at all. This is a different failure — the skill **was invoked**, but the response/tool-call args did not contain the `az quota` keyword because the agent used the MCP `azure-quota` tool instead of the Azure CLI.

## Root Cause Category

**Assertion mismatch**

## Diagnosis

### What was expected

The test asserts that the skill is invoked AND the response or tool call arguments include:
1. `"az quota"` (CLI command keyword) — in assistant message or tool call args
2. `"/subscriptions/"` (scope URI) — in assistant message or tool call args

````ts
// tests/azure-quotas/integration.test.ts:110
expect(isSkillUsed).toBe(true); // PASSED — skill was invoked
expect(mentionsQuotaCmd).toBe(true); // FAILED — "az quota" not found
expect(mentionsScope).toBe(true);
```

### What actually happened

The agent correctly invoked the `azure-quotas` skill and retrieved VM quota data for East US. However, it used the MCP `azure-quota` tool (not the Azure CLI `az quota` command):

```
tool: azure-quota
→ learn available commands
→ check VM quota usage in eastus
````

Since the agent used the MCP tool rather than issuing `az quota` CLI commands, the keyword `"az quota"` was absent from both the assistant message and tool call arguments. The assertion `mentionsQuotaCmd` evaluates to `false`.

### Why it failed

The `azure-quotas` skill (or agent behavior) prefers the `azure-quota` MCP tool over the `az quota` CLI when MCP is available. The test was written expecting the skill to produce `az quota` CLI commands, but the MCP path is a valid alternative. The test's keyword check doesn't cover MCP tool invocations.

### Suggested fix

**Option A (fix the skill):** Update `azure-quotas` skill instructions to always include `az quota` commands in the response alongside MCP tool usage.

**Option B (fix the test):** Update the assertion to also accept MCP tool invocations as a valid path:
```ts
const mentionsQuotaCmd = doesAssistantMessageIncludeKeyword(agentMetadata, "az quota")
|| doToolCallArgsIncludeKeyword(agentMetadata, "az quota")
|| isToolCalled(agentMetadata, "azure-quota"); // accept MCP tool
```

## azure-quotas Skill Invocation

| Skill | Invoked |
|-------|---------|
| **azure-quotas** | **Yes** |

## Skill Report Context

Per-test section from SKILL-REPORT.md (click to expand)

From artifact `integration-report-azure-quotas` (run #132):

- **Skill Invocation Success Rate:** 100% (10/10)
- **Overall Test Pass Rate:** 100% (14/14) ← per SKILL-REPORT (junit.xml shows 4/6 pass)
- **Average Confidence:** 97%

The skill was correctly invoked; quota data was successfully retrieved. The agent used MCP `azure-quota` tool exclusively, omitting `az quota` CLI syntax from the response.

## Environment

- **Runner OS:** ubuntu-latest
- **Node.js:** v24.x
- **Model:** claude-sonnet-4.5
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23848054525
- **Commit:** `e948e2993efd30da197ff1b0d3e0c43092e1f3d5`
- **Test file:** `tests/azure-quotas/integration.test.ts:110`

**Related:** #1450 (skill not invoked — a different root cause)

> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23854424383) · [◷](https://github.com/search?q=repo%3Amicrosoft%2FGitHub-Copilot-for-Azure+is%3Aissue+%22gh-aw-workflow-call-id%3A+microsoft%2FGitHub-Copilot-for-Azure%2Fanalyze-test-run%22&type=issues)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.