Azure Stack HCI edge modules consistently breach 6-hour GitHub Actions timeout during E2E validation

Check for previous/existing GitHub issues

• I have checked for previous/existing GitHub issues

Issue Type?

Bug/Feature Request (Hybrid)

Module Name

All Azure Stack HCI edge modules:

• avm/res/azure-stack-hci/virtual-machine-instance
• avm/res/azure-stack-hci/logical-network
• avm/res/azure-stack-hci/marketplace-gallery-image
• avm/res/azure-stack-hci/cluster

Description

Problem Statement

All Azure Stack HCI modules that deploy resources on top of an HCI cluster (VMI, logical-network, marketplace-gallery-image) consistently run 5h 30min – 5h 55min in E2E deployment validation, frequently breaching the 6-hour GitHub Actions hard limit. This causes CI failures completely unrelated to code quality and blocks PR merges.

Unlike the cluster module itself (tracked in #6775), these "edge" modules have an additional deployment layer — they must first create the full HCI cluster infrastructure, then deploy their module under test, then clean up. This makes them even more time-constrained than the cluster module alone.

---

Detailed Timing Analysis (VMI Module — Representative Example)

Successful run: #24930356710 — 5h 54min (6 min from timeout)

Phase-by-Phase Breakdown

Phase	Start (UTC)	End (UTC)	Duration	% of Total
Job setup + checkout	11:59	12:04	~5 min	1%
Host VM deployment	12:04	12:51	~47 min	13%
HCI cluster validation (56 steps)	12:51	13:54	~63 min	18%
HCI cluster deployment (56 steps)	13:54	16:27	~153 min	44%
Marketplace image provisioning	16:27	16:58	~31 min	9%
VMI deployment (module under test)	16:58	17:31	~33 min	9%
Resource cleanup (52 resources)	17:31	17:52	~21 min	6%
Total			~5h 53min	100%

Key Observation

The module under test (VMI) only takes ~33 minutes (9% of total time). The remaining 91% is shared infrastructure — host VM, HCI cluster creation, and cleanup. This ratio is similar for all HCI edge modules:

Module	Module Deploy Time	Total E2E Time	Infrastructure Overhead
VMI	~33 min	~5h 54min	91%
Logical-network	~5 min	~5h 30min	98%
Marketplace-gallery-image	~30 min	~5h 40min	91%
Cluster (standalone)	N/A	~5h 20min	100% (is the infra)

---

Cleanup Phase Detail

The AVM pipeline removes resources one-by-one in priority order (not RG-level delete). For HCI modules, this means removing 52 resources individually:

Cleanup Step	Duration	Bottleneck
Role assignments (11)	~1.5 min	—
Diagnostic settings	~10 sec	—
KeyVault delete + soft-delete purge	~10 min	Biggest bottleneck
Virtual Machine	~2 min	—
MSIs + Resource Group	~15 sec	—
Post-cleanup overhead	~7 min	—
Total	~21 min

Optimization applied: We've set enableSoftDelete: false on the test KeyVault (following the pattern used by avm/res/network/application-gateway and avm/res/web/serverfarm). This eliminates the ~10 min purge wait. PR forthcoming.

---

Why This Affects All HCI Edge Modules

Every HCI module that runs on an HCI cluster shares the same test dependency chain:

main.test.bicep
  └── dependencies.bicep (shared test harness)
        ├── hciHostDeployment.bicep    → Azure VM with nested Hyper-V (~47 min)
        │   ├── Stage 1-2: Install features + reboot
        │   ├── Stage 3-4: VHDX/AD/networking + reboot  
        │   └── Stage 5: DHCP + Hyper-V switches + Arc connection
        ├── ashciPrereqs.bicep         → KeyVault, storage, secrets (~5 min)
        └── HCI Cluster resource       → Cluster shell (~1 min)
  └── cluster module (br/public:avm/res/azure-stack-hci/cluster:0.4.0)
        └── ECE Validate + Deploy      → 56 steps (~3.5 hours)
  └── Module under test                → Actual test (~5-33 min)
  └── Cleanup                          → 52 resources (~21 min)

~5 hours of this pipeline is identical across all modules. Only the last 5-33 minutes differ.

---

AVM CI/CD Constraints That Make This Critical

Constraint	AVM Spec	Impact on HCI
6-hour hard limit	GitHub-hosted runners (ubuntu-latest)	Zero margin at 5h54m
Self-contained tests	No pre-provisioned resources	Must rebuild entire HCI env from scratch
Mandatory defaults + waf-aligned	BCPRMNFR1	2× full deployments per PR
Resource cleanup required	removeDeployment: true	21 min cleanup per test case
vCPU quota	Shared subscription (100 cores)	E48 host = max 2 parallel tests

---

Proposed Solutions (for Core Team Discussion)

Short-term (within current framework)

1. ✅ KeyVault soft-delete disabled — saves ~10 min per test (PR in progress)
2. Increase job timeout — if self-hosted runners or larger runners are available
3. Sequential test execution — run defaults first, cleanup, then waf-aligned (avoids quota issues)

Medium-term (requires core team support)

4. Pre-baked VM image (detailed in #6775) — eliminates ~40-55 min of host setup per run
5. Shared cluster infrastructure — deploy HCI cluster once, run multiple module tests against it, then cleanup
6. Parallel cleanup — remove resources in parallel instead of sequentially where safe

Long-term (architectural)

7. Persistent test environment — maintain a standing HCI cluster for validation (similar to how some Azure services maintain test beds)
8. Split validation — static validation + PSRule in one job, deployment in a separate longer-running job with extended timeout

---

Evidence: Upstream CI Also Failing

The upstream CI run triggered by the VMI PR merge is also hitting this timeout:

• Run: #24949247881 — both deploy jobs cancelled (still running after 6+ hours)
• This confirms the issue affects the upstream Azure/bicep-registry-modules CI, not just fork runs

---

Related Issues

• #6775 — HCI cluster module pre-baked image proposal (addresses host setup time)
• #6776 — HCI cluster deployment idempotency limitation

Additional Context

All timing data collected from actual CI runs on the fork chirag1603/bicep-registry-modules with Standard_E48bds_v5 host VMs (48 vCPU, 384GB RAM) in southeastasia region. The 6-hour limit is a hard constraint of GitHub Actions that cannot be configured.

Start with the HCI E2E harness named in the issue: main.test.bicep, dependencies.bicep, hciHostDeployment.bicep, and ashciPrereqs.bicep; compare its timing with the runs linked above. Review #6775 and the proposed cleanup, image, shared-infrastructure, and execution changes before selecting an approach. Done means the affected edge-module validations complete reliably within the six-hour GitHub Actions limit.