microsoft / microsoft/GitHub-Copilot-for-Azure

Integration test failure: azure-cost – skill invocation tests Timeout [Timeout]

Open
#1,657 0 comments 0 reactions 1 assignee Claimed by @taylorak View on GitHub
bug integration-test
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Prompts

Three skill-invocation tests consistently time out after 20 minutes (1200s Jest limit):

1. `"How can I reduce my Azure spending and find cost savings in my subscription?"` — 5 retries, avg ~200s each
2. `"Find orphaned and unused resources in my Azure subscription that I can delete"` — 5 retries, avg ~150-236s each
3. `"Find unused storage accounts to reduce my Azure costs"` — 1 run, 155s

## Summary

**Run:** [Integration Tests - all #135](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23899869938)
**Tests:**
- `azure-cost - Integration Tests › skill-invocation › invokes skill for cost savings prompt`
- `azure-cost - Integration Tests › skill-invocation › invokes skill for orphaned resources prompt`
- `azure-cost - Integration Tests › skill-invocation › invokes skill for storage cost optimization prompt`
**Result:** Fail (all 3 tests at exactly 1200.033s)
**Duration:** 20m 0s per test (Jest max timeout)

## Root Cause Category

**Timeout**

## Diagnosis

### What was expected

Each skill-invocation test runs the agent 5 times and expects all 5 to complete within the 1200s Jest timeout.

### What actually happened

The agent correctly invoked the `azure-cost` skill in all runs (100% invocation rate per SKILL-REPORT), but the tests exceeded the Jest timeout. These tests exercise real Azure Cost Management API calls with multiple retries and rate-limit backoff, causing each individual run to take 150–250 seconds. Running 5 retries sequentially exhausted the 20-minute limit.

From the SKILL-REPORT:
- Skill Invocation Rate: **100%** (all runs invoked azure-cost)
- These tests involve intensive Azure API queries: `az rest --method POST /providers/Microsoft.CostManagement/query` with resource-level pagination, orphaned resource detection, and rate-limit retry logic

### Why it failed

The cost savings and orphaned resources workflows involve extensive Azure API calls:
- Querying cost data across all resource groups
- Using Azure Resource Graph for orphaned resource detection
- Retrying 2-5 times per run due to API rate limiting (HTTP 429)
- Running `azqr` fallbacks when primary tools are unavailable

5 retries × ~200s per run = ~1000s, plus overhead, exceeds the 1200s timeout.

### Suggested fix

- Reduce retries from 5 to 3 for these long-running integration tests, OR
- Increase the Jest timeout for these specific tests (cost savings, orphaned resources, storage cost optimization) to 2400s, OR
- Split into single-run tests rather than 5-retry tests for the high-latency scenarios

## azure-cost Skill Invocation

| Skill | Invoked |
|-------|---------|
| **azure-cost** | **Yes** (100% in all runs; failure is timeout, not skill routing) |

## Skill Report Context

From `test-run-all-integration-azure-cost-SKILL-REPORT.md` (run #135):
- **Skill Invocation Success Rate:** 100% (64/64)
- **Overall Test Pass Rate:** 100% (per SKILL-REPORT internal tracking; JUnit shows 12/16 due to timeout)
- **Average Confidence:** 93%

## Environment

- **Runner OS:** ubuntu-latest
- **Node.js:** v24.x
- **Model:** claude-sonnet-4.6
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23899869938
- **Commit:** `dc149f6821c954cedf9045a0c88fcf7571f6b35c`
- **Test file:** `tests/azure-cost/integration.test.ts:138, :154, :202`

> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23906450280) · [◷](https://github.com/search?q=repo%3Amicrosoft%2FGitHub-Copilot-for-Azure+is%3Aissue+%22gh-aw-workflow-call-id%3A+microsoft%2FGitHub-Copilot-for-Azure%2Fanalyze-test-run%22&type=issues)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.