aws / aws/aws-cdk

core: synth crashes with EISDIR on cloud-placeholder directories in cdk.out

Open
#38,653 1 comment 0 reactions 1 assignee Claimed by @iankhou View on GitHub
@aws-cdk/core bug effort/medium p1 potential-regression
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

Since v2.262.0, `cdk synth` crashes with `EISDIR: illegal operation on a directory, read` on any project whose `cdk.out` contains a **directory reparse point that libuv cannot `readlink()`** — in my case Windows Cloud Files placeholders (`IO_REPARSE_TAG_CLOUD_2`, `0x9000201A`) created by Dropbox on stale `asset./` directories.

The cause is `hashFile()` in `packages/aws-cdk-lib/core/lib/private/synthesis-validation.ts`, which assumes "not a symlink" implies "a readable file":

```ts
function collectFilePaths(dir: string): string[] {
const results: string[] = [];
function walk(current: string) {
for (const entry of fs.readdirSync(current, { withFileTypes: true })) {
const full = path.join(current, entry.name);
if (entry.isDirectory()) {
walk(full);
} else if (entry.isFile() || entry.isSymbolicLink()) {
results.push(full); // <-- reparse-point directory lands here
}
}
}
walk(dir);
return results;
}

function hashFile(filePath: string): string {
const content = fs.lstatSync(filePath).isSymbolicLink()
? Buffer.from(fs.readlinkSync(filePath))
: fs.readFileSync(filePath); // <-- EISDIR: filePath is a directory
return crypto.createHash('sha256').update(content).digest('hex');
}
```

On Windows these two calls disagree about the same path:

* `readdirSync(..., { withFileTypes: true })` — libuv's `fs__scandir` checks `FILE_ATTRIBUTE_REPARSE_POINT` **before** `FILE_ATTRIBUTE_DIRECTORY`, so the entry is reported as `isSymbolicLink() === true`, `isDirectory() === false`. It is therefore never recursed into, and is pushed onto the file list.
* `lstatSync()` — libuv's `fs__readlink_handle` only resolves four tags (`IO_REPARSE_TAG_SYMLINK`, `IO_REPARSE_TAG_LX_SYMLINK`, `IO_REPARSE_TAG_MOUNT_POINT` with a drive-letter target, `IO_REPARSE_TAG_APPEXECLINK`). Anything else sets `ERROR_SYMLINK_NOT_SUPPORTED`, and `fs__stat_impl` then **retries with `do_lstat = 0`**, so the path is reported as an ordinary directory with `isSymbolicLink() === false`.

readdir says "link", lstat says "directory", and `readFileSync()` gets a directory.

The existing fix in #38299 does not cover this. It enumerates the tags libuv *can* `readlink()`, which structurally excludes cloud placeholders (`0x9000*`), ProjFS (`0x9000001C`), dedup (`0x80000013`), WCI (`0x80000018`), and volume-GUID junctions.

### Regression Issue

- [x] Select this option if this issue appears to be a regression.

### Last Known Working CDK Library Version

2.261.x

### Expected Behavior

`cdk synth` completes. The validation-plugin integrity check ignores filesystem entries it cannot meaningfully hash, rather than crashing synthesis.

### Current Behavior

```
Error: EISDIR: illegal operation on a directory, read
at Object.readSync (node:fs:798:18)
at Object.readSync (my_code\node_modules\.pnpm\aws-cdk-lib@2.263.0_constructs@10.8.1\node_modules\aws-cdk-lib\core\lib\private\perf.js:1:775)
at tryReadSync (node:fs:427:20)
at Object.readFileSync (node:fs:530:19)
at Object.readFileSync (my_code\node_modules\.pnpm\aws-cdk-lib@2.263.0_constructs@10.8.1\node_modules\aws-cdk-lib\core\lib\private\perf.js:1:775)
at hashFile (my_code\node_modules\.pnpm\aws-cdk-lib@2.263.0_constructs@10.8.1\node_modules\aws-cdk-lib\core\lib\private\synthesis-validation.js:8:5484)
at snapshotFileHashes (my_code\node_modules\.pnpm\aws-cdk-lib@2.263.0_constructs@10.8.1\node_modules\aws-cdk-lib\core\lib\private\synthesis-validation.js:8:4845)
at doInvokeValidationPlugins (my_code\node_modules\.pnpm\aws-cdk-lib@2.263.0_constructs@10.8.1\node_modules\aws-cdk-lib\core\lib\private\synthesis-validation.js:8:2741)
at validateTemplates (my_code\node_modules\.pnpm\aws-cdk-lib@2.263.0_constructs@10.8.1\node_modules\aws-cdk-lib\core\lib\private\synthesis-validation.js:1:5303)
at synthesize (my_code\node_modules\.pnpm\aws-cdk-lib@2.263.0_constructs@10.8.1\node_modules\aws-cdk-lib\core\lib\private\synthesis.js:1:3089) {
errno: -4068,
code: 'EISDIR',
syscall: 'read'
}
```

Every affected entry in `cdk.out` looks like this (`fsutil reparsepoint query`):

```
cdk.out\asset.940924c54ba4c1fef2fd24993625390b907a7cbf73f320bee32cb67b56a58d7a.lambda
lstat(): directory, isSymbolicLink() === false
readlinkSync(): throws EINVAL
Reparse Tag Value : 0x9000201a | Tag value: Microsoft | Tag value: Directory
LinkType: (none) Target: (none)
```

`0x9000201A` = `IO_REPARSE_TAG_CLOUD_2` — a Windows Cloud Files API directory placeholder. It has no link target; the reparse buffer holds a sync-engine identity blob.

### Reproduction Steps

Any directory inside `cdk.out` that is a reparse point with a tag libuv cannot `readlink()` will do. Two ways to produce one:

**1. With a cloud sync engine (what I hit).** Place a CDK project inside a OneDrive- or Dropbox-synced folder, `cdk synth`, and let the client sync `cdk.out`. It converts the `asset./` directories to placeholders in place via `CfConvertToPlaceholder`. Then `cdk synth` again. Note this is unrelated to files-on-demand/online-only — the reparse point is the sync engine's ownership marker and is present on fully-hydrated content.

**2. Without any third-party software**, using a volume mount point, which produces `IO_REPARSE_TAG_MOUNT_POINT` with a `\??\Volume{...}` substitute name that libuv also rejects (elevated prompt, needs any spare volume such as a USB stick):

```
mkdir cdk.out\asset.repro
mountvol cdk.out\asset.repro \\?\Volume{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}\
cdk synth
```

**Minimal unit-level invariant**, if that is easier to assert in a test: any path for which `readdirSync(withFileTypes)` reports `isSymbolicLink() === true` while `lstatSync()` reports `isDirectory() === true` and `isSymbolicLink() === false` will crash `hashFile()`.

### Possible Solution

Gate on what the entry *is*, rather than on what it is not. `lstat` is already being called, so this costs nothing extra:

```ts
function collectFilePaths(dir: string): string[] {
const results: string[] = [];
function walk(current: string) {
for (const entry of fs.readdirSync(current, { withFileTypes: true })) {
const full = path.join(current, entry.name);
if (entry.isDirectory()) {
walk(full);
} else if (entry.isFile() || entry.isSymbolicLink()) {
// readdir() classifies every reparse point as a link, but lstat() only agrees for the
// tags libuv can readlink(). Cloud placeholders, ProjFS, dedup, WCI and volume-GUID
// junctions come back as plain directories here; there is nothing to hash, and a
// validation plugin cannot meaningfully "modify" one, so skip them.
const st = fs.lstatSync(full);
if (st.isFile() || st.isSymbolicLink()) {
results.push(full);
}
}
}
}
walk(dir);
return results;
}
```

Since `collectFilePaths()` is the only producer of the paths that reach `hashFile()`, filtering there fixes both the initial snapshot and the `hasModifiedPreExistingFiles()` re-hash. `fileOrSymlinkExists()` already tolerates these entries.

Wrapping `hashFile()` in a `try/catch` that skips unreadable entries would also work as defence-in-depth. What should be avoided is another tag-by-tag allowlist — the set of reparse tags that libuv cannot resolve is open-ended and vendor-defined, so enumerating it will keep producing this bug.

### Additional Information/Context

Relevant commits:

* d9f38a9d381d8523a683c72a72395ade9837ea7e — feat(core): allow validation plugins to create new files in cloud assembly (#38007), first released in v2.258.0. Replaced the symlink-aware `FileSystem.fingerprint(outdir)` with the hand-rolled `collectFilePaths()` + `readFileSync()` walk.
* 023c5bf54816e5a0bdea355944d3db4a4d718bf9 — feat(core): templates are validated against a comprehensive default rule set (#38135), released in v2.262.0. Made the walk run for every app by registering `CloudFormationValidatePlugin` by default, and moved the code to `synthesis-validation.ts`. PR #38628 is there to fix/revert this change.
* be0fdc4c83 — fix(core): synth crashes with EISDIR on a symlink-to-directory when a validation plugin is registered (#38299), also in v2.262.0. Fixes the real-symlink case only.

libuv references (`src/win/fs.c`): `fs__scandir` for the readdir classification order, `fs__readlink_handle` for the four supported tags and its `SetLastError(ERROR_SYMLINK_NOT_SUPPORTED)` fallthrough, and `fs__stat_impl` for the `do_lstat = 0` retry.

Note that `cdk.out` is never cleaned by CDK, so stale `asset./` directories accumulate for years. That makes long-lived projects far more likely to contain an entry that some external tool has since stamped, even when a freshly synthesized `cdk.out` is clean — which is why this presents as "only my old projects are broken".

Workarounds for anyone else hitting this: delete `cdk.out` (it regenerates clean), move the assembly out of the synced tree with `"output"` in `cdk.json` or `CDK_OUTDIR`, or pin `aws-cdk-lib` below 2.262.0 if you register no validation plugins.

### AWS CDK Library version (aws-cdk-lib)

2.263.0

### AWS CDK CLI version

2.1137.0

### Node.js Version

v24.19.0

### OS

Windows 11

### Language

TypeScript

### Language Version

5.9.3

### Other information

Happens with Python too.

Bug report was written by Claude after I found and debugged the issue and found the root cause myself. Claude was used to add hopefully useful details, including history and workarounds.

Originally discussed on [Slack](https://cdk-dev.slack.com/archives/C018XT6REKT/p1785336955324009).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.