microsoft / microsoft/GitHub-Copilot-for-Azure

Integration test failure: azure-cost – VM rightsizing skill invocation below threshold [Skill not invoked]

Open
#1,981 0 comments 0 reactions 1 assignee Claimed by @joybb View on GitHub
bug integration-test test-failure
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Prompt

```
Rightsize my Azure VMs to reduce costs
```

## Summary

**Run:** [Integration Tests - all #176](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/24721935277)
**Test:** `azure-cost_ - Integration Tests › skill-invocation > invokes skill for VM rightsizing prompt`
**Result:** Fail (score 0.667, threshold ≥ 0.8 — 2/3 runs invoked the skill)
**Duration:** 143.3s

## Root Cause Category

**Skill not invoked**

## Diagnosis

### What was expected
The test runs the prompt 3 times and expects the `azure-cost` skill to be invoked at a rate ≥ 0.8 (i.e., in at least 3 of 3 runs, or 2.4+ runs).

### What actually happened
The `azure-cost` skill was invoked in only 2 of 3 runs (score = 0.667). In the run where it was invoked, both `azure-cost` and `azure-compute` skills were used, but the agent encountered API rate limiting (HTTP 429 from Azure Cost Management) and had to pivot to resource inventory analysis.

From the consolidated report:
- **Runs 1 & 2**: azure-cost invoked ✅
- **Run 3**: azure-cost not invoked ❌ (agent answered directly with general VM rightsizing advice without invoking the skill)
- API rate limiting (Cost Management API 429) forced the agent to use alternative analysis methods

### Why it failed
The prompt "Rightsize my Azure VMs to reduce costs" is borderline between a cost optimization query (azure-cost) and a compute management query (azure-compute). In 1 of 3 runs, the agent chose to answer from knowledge base without invoking azure-cost, possibly because the prompt mentions VMs specifically and the agent associates that with azure-compute rather than azure-cost.

### Suggested fix
- Add "VM rightsizing", "rightsize VMs", "resize VMs to save cost" as explicit trigger phrases in the `azure-cost` skill description
- Review whether the `azure-compute` skill description should explicitly redirect rightsizing-for-cost prompts to `azure-cost`

## azure-cost Skill Invocation

| Skill | Invoked |
|-------|---------|
| **azure-cost** | **Partial** — 2/3 runs (67%) |

## Skill Report Context

Per-test section from SKILL-REPORT.md (click to expand)

**Skill Invocation Success Rate:** 100% (33/33 across all other skill-invocation tests)
**Overall Test Pass Rate:** 97% (32/33)
**Average Confidence:** 90%

All other 15 test cases passed. The VM rightsizing prompt is the only test where skill routing was inconsistent.

## Environment

- **Runner OS:** ubuntu-latest
- **Model:** claude-sonnet-4.5
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/24721935277
- **Commit:** `21aedb8d6699465050abf56eb68cd8a2fd13ca4d`
- **Test file:** `tests/azure-cost/integration.test.ts:183`

> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/24730438080)

> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/24730438080/agentic_workflow) · ● 3.1M · [◷](https://github.com/search?q=repo%3Amicrosoft%2FGitHub-Copilot-for-Azure+is%3Aissue+%22gh-aw-workflow-call-id%3A+microsoft%2FGitHub-Copilot-for-Azure%2Fanalyze-test-run%22&type=issues)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.