aws / aws/aws-cdk

asset staging: do not recurse into the output directory

Open
#3,899 20 comments 35 reactions 0 assignees View on GitHub
@aws-cdk/core bug effort/small p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

## :bug: Bug Report

### What is the problem?

I have hit a corner case with the code related to copy directories. It's going in recursively until Node can no longer handle such long file names.

https://github.com/aws/aws-cdk/blob/16eb65867b79798779bf98a0316b1a05915720f1/packages/%40aws-cdk/assets/lib/fs/copy.ts#L7-L18

The offending snippet is when I had a docker asset declared as follows:
```javascript
new DockerImageAsset(this, 'app-image-asset', {
directory: '.',
repositoryName: 'my-repo/app',
});
```

Running `cdk deploy` showed me this error:
```
Error: ENAMETOOLONG: name too long, mkdir 'cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.03918990104d8973acd2ad0abe022844192fcce618cf74f7734a7077f4111d1f'
at Object.mkdirSync (fs.js:750:3)
at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:39:16)
at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
```

_(Declaring the directory as its full path did not work neither.)_
This is quite a serious issue for me, personally.

A possible solution is to exclude `cdk.out`. I think it's reasonable to assume that `cdk.out` name is reserved for this project.

As a temporary workaround for me, I've edited the code so that it looks like the following:
```javascript
const exclude = options.exclude || ['cdk.out'];
```

Another possible solution is to avoid recursion if we've already seen the directory in question.

Apparently, it's an issue where the docker build context is the same path as the cdk project.
That said, I'm not sure if the above is enough to reproduce the issue. When I comment the code to create the docker asset, the issue disappears (probably more of a symptom rather than the cause). I've checked if symlinks are the issue (apparently, it's not)

### Environment

- **CDK CLI Version:** 1.6.1
- **OS:** all
- **Language:** all

Contributor guide

Open the contributing guide

Research direction

Start with packages/@aws-cdk/assets/lib/fs/copy.ts, especially the copyDirectory logic linked in the report. Reproduce with a DockerImageAsset using directory '.' and cdk deploy, then verify that staging does not recurse into cdk.out and avoids the reported ENAMETOOLONG failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, typescript
Domain
build-system, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.