microsoft / microsoft/GitHub-Copilot-for-Azure

Integration test failure: azure-quotas – region comparison CLI keyword missing when using MCP tool [Assertion mismatch]

Open
#1,627 0 comments 0 reactions 1 assignee Claimed by @rakal-dyh View on GitHub
azure-quotas bug integration-test test-failure
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Prompt

```
Compare Azure compute quota availability across East US, West US 2, and Central US
```

## Summary

**Run:** [Integration Tests - all #132](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23848054525)
**Test:** `azure-quotas_ - Integration Tests › azure-quotas › handles region comparison query`
**Result:** Fail
**Duration:** 30.1s

## Root Cause Category

**Assertion mismatch**

## Diagnosis

### What was expected

The test asserts that the skill is invoked AND the response or tool call arguments include `"az quota"`:

```ts
// tests/azure-quotas/integration.test.ts:141
expect(isSkillUsed).toBe(true); // PASSED — skill was invoked
expect(mentionsQuotaCmd).toBe(true); // FAILED — "az quota" not found
```

Where:
````ts
const mentionsQuotaCmd = doesAssistantMessageIncludeKeyword(agentMetadata, "az quota")
|| doToolCallArgsIncludeKeyword(agentMetadata, "az quota");
```

### What actually happened

The agent correctly invoked the `azure-quotas` skill and successfully compared quota availability across all three regions (East US, West US 2, Central US). It used the MCP `azure-quota` tool (4 calls) — one to learn available commands, then one per region. The quota data was retrieved and a comprehensive comparison was provided.

However, because the agent exclusively used the MCP `azure-quota` tool rather than issuing `az quota` CLI commands, the string `"az quota"` did not appear in the assistant message or tool call arguments. The keyword assertion failed.

**Agent tool usage:**
```
tool: azure-quota → learn commands
tool: azure-quota → quota_usage_check eastus (100+ VM SKUs)
tool: azure-quota → quota_usage_check westus2
tool: azure-quota → quota_usage_check centralus
````

### Why it failed

Same root cause as the `provides quota check commands for compute resources` test (see #1628): the agent prefers the `azure-quota` MCP tool over the `az quota` CLI, and the test assertion does not account for the MCP execution path.

### Suggested fix

**Option A (fix the skill):** Update `azure-quotas` skill instructions to include `az quota` CLI commands in the response for transparency, even when MCP tool is used.

**Option B (fix the test):** Update the assertion to accept MCP tool invocations:
```ts
const mentionsQuotaCmd = doesAssistantMessageIncludeKeyword(agentMetadata, "az quota")
|| doToolCallArgsIncludeKeyword(agentMetadata, "az quota")
|| isToolCalled(agentMetadata, "azure-quota"); // accept MCP tool path
```

## azure-quotas Skill Invocation

| Skill | Invoked |
|-------|---------|
| **azure-quotas** | **Yes** |

## Skill Report Context

Per-test section from SKILL-REPORT.md (click to expand)

From artifact `integration-report-azure-quotas` (run #132):

- **Skill Invocation Success Rate:** 100% (10/10)
- **Average Confidence:** 97%

The agent retrieved complete quota data for all 3 regions, identified GPU/specialty compute requiring quota increases, and produced a regional comparison. The MCP tool (`azure-quota`) was used exclusively.

## Environment

- **Runner OS:** ubuntu-latest
- **Node.js:** v24.x
- **Model:** claude-sonnet-4.5
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23848054525
- **Commit:** `e948e2993efd30da197ff1b0d3e0c43092e1f3d5`
- **Test file:** `tests/azure-quotas/integration.test.ts:141`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.