microsoft / microsoft/GitHub-Copilot-for-Azure
Integration test failure: azure-cost – skill invocation tests Timeout [Timeout]
- Dominant language
- Python
- Stars
- 250
- Forks
- 204
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 67
Description
## Prompts
Three skill-invocation tests consistently time out after 20 minutes (1200s Jest limit):
1. `"How can I reduce my Azure spending and find cost savings in my subscription?"` — 5 retries, avg ~200s each
2. `"Find orphaned and unused resources in my Azure subscription that I can delete"` — 5 retries, avg ~150-236s each
3. `"Find unused storage accounts to reduce my Azure costs"` — 1 run, 155s
## Summary
**Run:** [Integration Tests - all #135](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23899869938)
**Tests:**
- `azure-cost - Integration Tests › skill-invocation › invokes skill for cost savings prompt`
- `azure-cost - Integration Tests › skill-invocation › invokes skill for orphaned resources prompt`
- `azure-cost - Integration Tests › skill-invocation › invokes skill for storage cost optimization prompt`
**Result:** Fail (all 3 tests at exactly 1200.033s)
**Duration:** 20m 0s per test (Jest max timeout)
## Root Cause Category
**Timeout**
## Diagnosis
### What was expected
Each skill-invocation test runs the agent 5 times and expects all 5 to complete within the 1200s Jest timeout.
### What actually happened
The agent correctly invoked the `azure-cost` skill in all runs (100% invocation rate per SKILL-REPORT), but the tests exceeded the Jest timeout. These tests exercise real Azure Cost Management API calls with multiple retries and rate-limit backoff, causing each individual run to take 150–250 seconds. Running 5 retries sequentially exhausted the 20-minute limit.
From the SKILL-REPORT:
- Skill Invocation Rate: **100%** (all runs invoked azure-cost)
- These tests involve intensive Azure API queries: `az rest --method POST /providers/Microsoft.CostManagement/query` with resource-level pagination, orphaned resource detection, and rate-limit retry logic
### Why it failed
The cost savings and orphaned resources workflows involve extensive Azure API calls:
- Querying cost data across all resource groups
- Using Azure Resource Graph for orphaned resource detection
- Retrying 2-5 times per run due to API rate limiting (HTTP 429)
- Running `azqr` fallbacks when primary tools are unavailable
5 retries × ~200s per run = ~1000s, plus overhead, exceeds the 1200s timeout.
### Suggested fix
- Reduce retries from 5 to 3 for these long-running integration tests, OR
- Increase the Jest timeout for these specific tests (cost savings, orphaned resources, storage cost optimization) to 2400s, OR
- Split into single-run tests rather than 5-retry tests for the high-latency scenarios
## azure-cost Skill Invocation
| Skill | Invoked |
|-------|---------|
| **azure-cost** | **Yes** (100% in all runs; failure is timeout, not skill routing) |
## Skill Report Context
From `test-run-all-integration-azure-cost-SKILL-REPORT.md` (run #135):
- **Skill Invocation Success Rate:** 100% (64/64)
- **Overall Test Pass Rate:** 100% (per SKILL-REPORT internal tracking; JUnit shows 12/16 due to timeout)
- **Average Confidence:** 93%
## Environment
- **Runner OS:** ubuntu-latest
- **Node.js:** v24.x
- **Model:** claude-sonnet-4.6
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23899869938
- **Commit:** `dc149f6821c954cedf9045a0c88fcf7571f6b35c`
- **Test file:** `tests/azure-cost/integration.test.ts:138, :154, :202`
> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23906450280) · [◷](https://github.com/search?q=repo%3Amicrosoft%2FGitHub-Copilot-for-Azure+is%3Aissue+%22gh-aw-workflow-call-id%3A+microsoft%2FGitHub-Copilot-for-Azure%2Fanalyze-test-run%22&type=issues)
Contributor guide
Assessment
This issue has not been assessed yet.