microsoft / microsoft/GitHub-Copilot-for-Azure

Integration test failure: azure-quotas – quota check VM compute resources skill not invoked [Skill not invoked]

Open
#1,450 0 comments 0 reactions 1 assignee Claimed by @rakal-dyh View on GitHub
azure-quotas bug integration-test test-failure
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Prompt

````
Check my Azure VM quota limits and current usage in East US
```

## Summary

**Run:** [Integration Tests - all #110](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23379277272)
**Test:** `azure-quotas_ - Integration Tests › azure-quotas › provides quota check commands for compute resources`
**Result:** Fail
**Duration:** 5m 50s

## Root Cause Category

**Skill not invoked**

## Diagnosis

### What was expected
The test expects all three of:
1. `isSkillUsed` (= `isSkillInvoked(agentMetadata, SKILL_NAME)`) = `true`
2. `mentionsQuotaCmd` = the agent response or tool call args include `az quota`
3. `mentionsScope` = the agent response or tool call args include `/subscriptions/`

### What actually happened
`isSkillUsed` was `false` — the `azure-quotas` skill was not invoked. The subsequent assertions (`mentionsQuotaCmd`, `mentionsScope`) also likely failed as a result.

```
> 115 | expect(isSkillUsed).toBe(true);
````

### Why it failed
The prompt "Check my Azure VM quota limits and current usage in East US" did not trigger the `azure-quotas` skill. The agent may have answered the quota question from general knowledge or used Azure CLI commands directly without routing through the skill. The skill's description may not strongly match "check quota limits and current usage" queries.

### Suggested fix
- Update `azure-quotas` skill description to include trigger phrases like "quota limits", "current usage", "VM quota", "vCPU limits"
- Verify the skill's "When to Use" section covers compute quota check queries by region
- Test whether the prompt is reaching the skill in live Copilot sessions

## azure-quotas Skill Invocation

| Skill | Invoked |
|-------|---------|
| **azure-quotas** | No |

## Skill Report Context

Skill report artifact not accessible (network policy blocks Azure Blob Storage downloads in the AWF sandbox). See run artifacts at https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23379277272/artifacts/6039510637

## Environment

- **Runner OS:** ubuntu-latest
- **Node.js:** v24.x
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23379277272
- **Commit:** `2f186f53859a62d458f62897cf089476ca1eb05d`
- **Test file:** `azure-quotas/integration.test.ts:115`

> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23381600951) · [◷](https://github.com/search?q=repo%3Amicrosoft%2FGitHub-Copilot-for-Azure+is%3Aissue+%22gh-aw-workflow-call-id%3A+microsoft%2FGitHub-Copilot-for-Azure%2Fanalyze-test-run%22&type=issues)

> [!WARNING]
>
> ⚠️ Firewall blocked 1 domain
>
> The following domain was blocked by the firewall during workflow execution:
>
> - `productionresultssa3.blob.core.windows.net`
>
> To allow these domains, add them to the `network.allowed` list in your workflow frontmatter:
>
> ```yaml
> network:
> allowed:
> - defaults
> - "productionresultssa3.blob.core.windows.net"
> ```
>
> See [Network Configuration](https://github.github.com/gh-aw/reference/network/) for more information.
>
>

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.