core: CloudFormationValidatePlugin hangs indefinitely on Azure DevOps hosted agents
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
After upgrading from aws-cdk-lib 2.261.0 to 2.262.1, cdk synth hangs indefinitely on Azure DevOps hosted agents (ubuntu-latest). The same synth completes in ~30 seconds locally on the same codebase.
The hang is caused by the new CloudFormationValidatePlugin (WASM-based CloudFormation linter) introduced in #38135, which runs automatically during synthesis in 2.262.0+.
Setting CDK_VALIDATION=false resolves the hang — synth completes in ~6 minutes on the same hosted agent.
### Regression Issue
- [x] Select this option if this issue appears to be a regression.
### Last Known Working CDK Library Version
2.261.0
### Expected Behavior
cdk synth should complete within a reasonable time on CI hosted agents, or the validation plugin should have a timeout/graceful fallback.
### Current Behavior
cdk synth produces initial output (deprecation warnings, bundling), then hangs with no further output until the Azure DevOps worker timeout kills the job after ~60 minutes.
### Reproduction Steps
1. CDK app with ~20 stacks (TypeScript)
2. Upgrade aws-cdk-lib from 2.261.0 to 2.262.1
3. Run cdk synth on an Azure DevOps hosted agent (ubuntu-latest)
4. Observe: synth hangs after deprecation warnings, never completes
### Possible Solution
The CloudFormationValidatePlugin (WASM-based linter) should either:
- Be opt-in rather than opt-out (require CDK_VALIDATION=true to enable)
- Have a timeout that gracefully skips validation if it takes too long
- Or investigate why the WASM engine (@aws/cloudformation-validate RegoEngine) hangs on resource-constrained CI environments while working fine locally
### Additional Information/Context
- Locally (Docker Desktop, same Node version, same codebase): synth completes in 30 seconds with validation enabled
- On hosted agent with CDK 2.261.0: synth completes in 5 minutes
- On hosted agent with CDK 2.262.1 + CDK_VALIDATION=false: synth completes in 6 minutes
- On hosted agent with CDK 2.262.1 without the env var: hangs >60 minutes
- The app has ~20 stacks
- The hosted agent is ubuntu-latest (vmImage 20260720.247.2) with default resource allocation
- The issue may be related to WASM execution performance or memory constraints on shared CI infrastructure
- Introduced by PR #38135
### AWS CDK Library version (aws-cdk-lib)
2.262.1
### AWS CDK CLI version
2.1129.0
### Node.js Version
24.18.0
### OS
Ubuntu 24.04.4 LTS (Azure DevOps hosted agent, vmImage ubuntu-latest)
### Language
TypeScript
### Language Version
5.9.3
### Other information
Workaround: set CDK_VALIDATION=false as environment variable before running cdk synth.
The plugin is registered by default via defaultValidationEnabled() in packages/aws-cdk-lib/core/lib/private/synthesis-validation.js, which only checks process.env.CDK_VALIDATION !== "false".
This affects CI environments with Docker-based bundling (Python Lambda layers), but may also affect apps without bundling — we did not test exhaustively. The common factor is running on Azure DevOps hosted agents with default resource allocation.
Related: https://github.com/aws/aws-cdk/pull/38135
Contributor guide
Research direction
Start with packages/aws-cdk-lib/core/lib/private/synthesis-validation.js and the validation-plugin changes from PR #38135. Reproduce cdk synth on an Azure DevOps hosted agent with validation enabled and disabled, then compare the plugin's behavior and timing. Done means synthesis no longer hangs indefinitely in the reported CI environment while preserving validation behavior where it completes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript, wasm
- Domain
- build-system, devops, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100