aws / aws/aws-cdk

core: CloudFormationValidatePlugin hangs indefinitely on Azure DevOps hosted agents

Open
#38,425 4 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/core bug cloudformation-validation p2 potential-regression
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

After upgrading from aws-cdk-lib 2.261.0 to 2.262.1, cdk synth hangs indefinitely on Azure DevOps hosted agents (ubuntu-latest). The same synth completes in ~30 seconds locally on the same codebase.

The hang is caused by the new CloudFormationValidatePlugin (WASM-based CloudFormation linter) introduced in #38135, which runs automatically during synthesis in 2.262.0+.

Setting CDK_VALIDATION=false resolves the hang — synth completes in ~6 minutes on the same hosted agent.

### Regression Issue

- [x] Select this option if this issue appears to be a regression.

### Last Known Working CDK Library Version

2.261.0

### Expected Behavior

cdk synth should complete within a reasonable time on CI hosted agents, or the validation plugin should have a timeout/graceful fallback.

### Current Behavior

cdk synth produces initial output (deprecation warnings, bundling), then hangs with no further output until the Azure DevOps worker timeout kills the job after ~60 minutes.

### Reproduction Steps

1. CDK app with ~20 stacks (TypeScript)
2. Upgrade aws-cdk-lib from 2.261.0 to 2.262.1
3. Run cdk synth on an Azure DevOps hosted agent (ubuntu-latest)
4. Observe: synth hangs after deprecation warnings, never completes

### Possible Solution

The CloudFormationValidatePlugin (WASM-based linter) should either:
- Be opt-in rather than opt-out (require CDK_VALIDATION=true to enable)
- Have a timeout that gracefully skips validation if it takes too long
- Or investigate why the WASM engine (@aws/cloudformation-validate RegoEngine) hangs on resource-constrained CI environments while working fine locally

### Additional Information/Context

- Locally (Docker Desktop, same Node version, same codebase): synth completes in 30 seconds with validation enabled
- On hosted agent with CDK 2.261.0: synth completes in 5 minutes
- On hosted agent with CDK 2.262.1 + CDK_VALIDATION=false: synth completes in 6 minutes
- On hosted agent with CDK 2.262.1 without the env var: hangs >60 minutes
- The app has ~20 stacks
- The hosted agent is ubuntu-latest (vmImage 20260720.247.2) with default resource allocation
- The issue may be related to WASM execution performance or memory constraints on shared CI infrastructure
- Introduced by PR #38135

### AWS CDK Library version (aws-cdk-lib)

2.262.1

### AWS CDK CLI version

2.1129.0

### Node.js Version

24.18.0

### OS

Ubuntu 24.04.4 LTS (Azure DevOps hosted agent, vmImage ubuntu-latest)

### Language

TypeScript

### Language Version

5.9.3

### Other information

Workaround: set CDK_VALIDATION=false as environment variable before running cdk synth.

The plugin is registered by default via defaultValidationEnabled() in packages/aws-cdk-lib/core/lib/private/synthesis-validation.js, which only checks process.env.CDK_VALIDATION !== "false".

This affects CI environments with Docker-based bundling (Python Lambda layers), but may also affect apps without bundling — we did not test exhaustively. The common factor is running on Azure DevOps hosted agents with default resource allocation.

Related: https://github.com/aws/aws-cdk/pull/38135

Contributor guide

Open the contributing guide

Research direction

Start with packages/aws-cdk-lib/core/lib/private/synthesis-validation.js and the validation-plugin changes from PR #38135. Reproduce cdk synth on an Azure DevOps hosted agent with validation enabled and disabled, then compare the plugin's behavior and timing. Done means synthesis no longer hangs indefinitely in the reported CI environment while preserving validation behavior where it completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript, wasm
Domain
build-system, devops, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.