microsoft / microsoft/winget-pkgs

[Pipeline Bug]: Azure Setup fails with transient networking errors — pipeline should auto-retry

Open
#379,018 0 comments 1 reaction 0 assignees View on GitHub
Area-Validation-Pipeline Issue-Feature
Dominant language
No language data
Stars
11.1k
Forks
9.7k
Avg merge
1h 32m
Merged PRs (30d)
616

Description

### Description of the new feature/enhancement

The "Azure Setup" step in the validation pipeline fails intermittently due to transient networking issues on the agent VMs. These failures block PR validation and require manual retries, creating unnecessary delays for contributors.

**Observed failure patterns (May 21–23, 2026):**

| Error Pattern | Frequency | Example Build |
|---------------|-----------|---------------|
| Exit code 0xDEAD (57005) — `az cloud set` hangs and node process is killed after ~18s | 3 of 5 | [Build 325357](https://dev.azure.com/shine-oss/winget-pkgs/_build/results?buildId=325357) |
| `connect ETIMEDOUT 150.171.73.16:443` — TCP timeout to Azure Management endpoint | 1 of 5 | [Build 324872](https://dev.azure.com/shine-oss/winget-pkgs/_build/results?buildId=324872) |
| `Failed to resolve 'login.microsoftonline.com'` — DNS resolution failure | 1 of 5 | [Build 324473](https://dev.azure.com/shine-oss/winget-pkgs/_build/results?buildId=324473) |

**Affected PRs:**
- https://github.com/microsoft/winget-pkgs/pull/373715
- https://github.com/microsoft/winget-pkgs/pull/374624
- https://github.com/microsoft/winget-pkgs/pull/376196
- https://github.com/microsoft/winget-pkgs/pull/376662
- https://github.com/microsoft/winget-pkgs/pull/378423

All failures are transient infrastructure issues — not caused by PR content.

### Proposed technical implementation details (optional)

The pipeline should automatically retry when the Azure Setup step fails with known transient error patterns:

1. **Detect transient failure:** After "Azure Setup" fails, check the exit code and error output:
- Exit code `57005` (0xDEAD) → transient timeout
- Error contains `ETIMEDOUT` → transient network timeout
- Error contains `getaddrinfo failed` or `Failed to resolve` → transient DNS failure

2. **Auto-retry with backoff:** Re-queue the validation run (up to 2 retries) with exponential backoff (e.g., 30s, 60s) before marking the PR as failed.

3. **Label differentiation:** If all retries are exhausted, apply a distinct label (e.g., `Internal-Error-Transient-Network`) rather than the generic `Internal-Error-Dynamic-Scan` so these can be distinguished from real validation failures.

This would eliminate manual retry overhead for contributors and maintainers, and reduce noise in the PR queue.

Contributor guide

Open the contributing guide

Research direction

Start with the validation pipeline's Azure Setup step and the linked build logs for builds 325357, 324872, and 324473. Trace how exit code 57005, ETIMEDOUT, and DNS resolution failures are reported, then verify that transient failures receive up to two backoff retries and exhausted retries use the distinct label.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
ci-cd, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.