aws / aws/aws-cdk

core: CloudFormationValidatePlugin tamper-detection hashes entire cdk.out — O(outdir size) per synth, ~225s stall on long-lived working dirs

Open
#38,614 1 comment 0 reactions 0 assignees View on GitHub
@aws-cdk/core effort/medium p1 potential-regression
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

Since the built-in `CloudFormationValidatePlugin` was introduced (2.262.0, #38135), synthesis-time validation in `core/lib/private/synthesis-validation.js` snapshots the assembly for tamper detection by SHA-256 hashing **every file under the assembly outdir** before validation (`snapshotFileHashes` / `collectFilePaths`) and `lstat`-ing them all again afterwards.

The assembly outdir is typically `cdk.out`, which on long-lived working directories accumulates staged `asset.*` directories from every prior deploy — CDK never prunes them. The tamper-detection snapshot is therefore O(total size of cdk.out), not O(size of the current synth output), and the cost grows unboundedly with every deploy into the same outdir.

On our app, a `cdk.out` that had accumulated 11 GB / 1,247,290 files across ~100 stale `asset.*` staging directories made every `cdk deploy` stall for ~225 seconds in `validateTemplates`, with no output and no indication of what it was doing. The actual Rego validation itself took under a second — virtually all of the time is hashing/stat-ing stale staged assets that have nothing to do with the templates being validated.

`cdk.out/performance-counters.json` from an affected synth (times in ms):

```json
{
"phase:Synthesis": 213674,
"validateTemplates": 213623,
"fs.readFileSync": 140534,
"fs.readFileSync(cnt)": 2494878,
"fs.lstatSync": 40236,
"fs.lstatSync(cnt)": 3744330,
"fs.readdirSync": 12875,
"fs.readdirSync(cnt)": 121754,
"CloudFormationValidate.validate": 769,
"CloudFormationValidate.validate(cnt)": 2
}
```

2.49M `readFileSync` calls and 3.74M `lstatSync` calls for a synth that validates two templates. Note `CloudFormationValidate.validate` itself is 769 ms — the other ~213 s is the file-hash snapshot walking stale assets.

Verified isolations:
- Same app, `cdk synth --output ` with validation ON: **3.3 s**.
- Same app, same dirty `cdk.out`, `--no-validation`: **2.3 s**.
- Deleting `cdk.out` fixes it — until staged assets accumulate again.

Possibly related: #38425 reports the validator hanging on CI agents; that appears to be the Rego engine itself, whereas this report is specifically the O(outdir-size) tamper-detection file hashing.

### Regression Issue

- [x] Select this option if this issue appears to be a regression.

### Last Known Working CDK Library Version

2.261.0 (before the built-in validator)

### Expected Behavior

Synthesis-time validation overhead should be proportional to the templates being validated, not to the historical size of the output directory. Users with long-lived working dirs (the default local workflow — `cdk.out` is normally gitignored and never cleaned) should not see multi-minute silent stalls.

### Current Behavior

Every synth/deploy silently stalls for minutes in `validateTemplates` while the tamper-detection snapshot hashes and stats every stale `asset.*` staging directory ever written to `cdk.out`. There is no progress output, so it looks like a hang.

### Reproduction Steps

1. Any CDK app with assets (e.g. a Lambda with `Code.fromAsset`), aws-cdk-lib >= 2.262.0.
2. Deploy repeatedly over time so `cdk.out` accumulates many `asset.*` staging dirs (or simulate: copy a few GB of files into `cdk.out/asset.fake1/...` etc.).
3. Run `cdk synth`.
4. Observe `validateTemplates` time in `cdk.out/performance-counters.json` scale with total `cdk.out` size, dominated by `fs.readFileSync`/`fs.lstatSync` counts.

### Possible Solution

- Restrict the tamper-detection snapshot to files that are part of the current cloud assembly (manifest-listed templates/artifacts), not everything under the outdir — stale `asset.*` dirs from prior runs cannot be tampered with by a validation plugin acting on this synth.
- Or skip hashing asset staging directories entirely (hash templates + manifest only).
- Or hash lazily/only when at least one third-party validation plugin is registered — the built-in plugin presumably doesn't need to be defended against itself.

### Additional Information/Context

Workaround we're using: wipe `cdk.out` before each deploy (it's fully regenerated). `--no-validation` also works but throws away the validator.

### AWS CDK Library version (aws-cdk-lib)

2.265.0

### AWS CDK CLI version

2.1138.0

### Node.js Version

22.x

### OS

macOS 15 (Darwin 24.6.0)

### Language

JavaScript/TypeScript

Contributor guide

Open the contributing guide

Research direction

Start in core/lib/private/synthesis-validation.js, reading snapshotFileHashes and collectFilePaths to trace why validation walks the entire cdk.out. Reproduce with a long-lived output directory and compare its performance counters with a fresh directory. Done means validation no longer hashes or stats stale asset staging files while still covering the current assembly used for validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.