microsoft / microsoft/GitHub-Copilot-for-Azure
Integration test failure: azure-cost – VM rightsizing skill invocation below threshold [Skill not invoked]
- Dominant language
- Python
- Stars
- 250
- Forks
- 204
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 67
Description
## Prompt
```
Rightsize my Azure VMs to reduce costs
```
## Summary
**Run:** [Integration Tests - all #176](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/24721935277)
**Test:** `azure-cost_ - Integration Tests › skill-invocation > invokes skill for VM rightsizing prompt`
**Result:** Fail (score 0.667, threshold ≥ 0.8 — 2/3 runs invoked the skill)
**Duration:** 143.3s
## Root Cause Category
**Skill not invoked**
## Diagnosis
### What was expected
The test runs the prompt 3 times and expects the `azure-cost` skill to be invoked at a rate ≥ 0.8 (i.e., in at least 3 of 3 runs, or 2.4+ runs).
### What actually happened
The `azure-cost` skill was invoked in only 2 of 3 runs (score = 0.667). In the run where it was invoked, both `azure-cost` and `azure-compute` skills were used, but the agent encountered API rate limiting (HTTP 429 from Azure Cost Management) and had to pivot to resource inventory analysis.
From the consolidated report:
- **Runs 1 & 2**: azure-cost invoked ✅
- **Run 3**: azure-cost not invoked ❌ (agent answered directly with general VM rightsizing advice without invoking the skill)
- API rate limiting (Cost Management API 429) forced the agent to use alternative analysis methods
### Why it failed
The prompt "Rightsize my Azure VMs to reduce costs" is borderline between a cost optimization query (azure-cost) and a compute management query (azure-compute). In 1 of 3 runs, the agent chose to answer from knowledge base without invoking azure-cost, possibly because the prompt mentions VMs specifically and the agent associates that with azure-compute rather than azure-cost.
### Suggested fix
- Add "VM rightsizing", "rightsize VMs", "resize VMs to save cost" as explicit trigger phrases in the `azure-cost` skill description
- Review whether the `azure-compute` skill description should explicitly redirect rightsizing-for-cost prompts to `azure-cost`
## azure-cost Skill Invocation
| Skill | Invoked |
|-------|---------|
| **azure-cost** | **Partial** — 2/3 runs (67%) |
## Skill Report Context
Per-test section from SKILL-REPORT.md (click to expand)
**Skill Invocation Success Rate:** 100% (33/33 across all other skill-invocation tests)
**Overall Test Pass Rate:** 97% (32/33)
**Average Confidence:** 90%
All other 15 test cases passed. The VM rightsizing prompt is the only test where skill routing was inconsistent.
## Environment
- **Runner OS:** ubuntu-latest
- **Model:** claude-sonnet-4.5
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/24721935277
- **Commit:** `21aedb8d6699465050abf56eb68cd8a2fd13ca4d`
- **Test file:** `tests/azure-cost/integration.test.ts:183`
> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/24730438080)
> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/24730438080/agentic_workflow) · ● 3.1M · [◷](https://github.com/search?q=repo%3Amicrosoft%2FGitHub-Copilot-for-Azure+is%3Aissue+%22gh-aw-workflow-call-id%3A+microsoft%2FGitHub-Copilot-for-Azure%2Fanalyze-test-run%22&type=issues)
Contributor guide
Assessment
This issue has not been assessed yet.