aws / aws/aws-cdk

(custom-resources): empty onEvent handler zip's being created, failing deploys

Open
#27,342 22 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/custom-resources bug p1
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

We recently started to see our integration tests failing, even though deploys were succeeding. The failures on the integration tests look like this:

```
sent 1,788 bytes received 35 bytes 3,646.00 bytes/sec
total size is 1,680 speedup is 0.92
fatal: Not a valid object name integ
INFRA-MYAPP-ClusterTest: fail: ENOENT: no such file or directory, open '/home/runner/work/x/xxx/test/integ/constructs/xyz/integ.cluster.ts.snapshot/asset.9202bb21d52e07810fc1da0f6acf2dcb75a40a43a9a2efbcfc9ae39535c6260c.zip'
INFRA-MYAPP-ClusterTest: fail: ENOENT: no such file or directory, open '/home/runner/work/xxx/xxx/test/integ/constructs/xyz/integ.cluster.ts.snapshot/asset.8e18eb5caccd2617fb76e648fa6a35dc0ece98c4681942bc6861f41afdff6a1b.zip'
INFRA-MYAPP-ClusterTest: fail: ENOENT: no such file or directory, open '/home/runner/work/xxx/xxx/test/integ/constructs/xyz/integ.cluster.ts.snapshot/asset.e2277687077a2abf9ae1af1cc9565e6715e2ebb62f79ec53aa75a1af9298f642.zip'

❌ Deployment failed: Error: Failed to publish asset a3f66c60067b06b5d9d00094e9e817ee39dd7cb5c315c8c254f5f3c571959ce5:current_account-current_region
at Deployments.publishSingleAsset (/home/runner/work/xxx/xxx/node_modules/aws-cdk/lib/index.js:446:11458)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Object.publishAsset (/home/runner/work/xxx/xxx/node_modules/aws-cdk/lib/index.js:446:151474)
at async /home/runner/work/xxx/xxx/node_modules/aws-cdk/lib/index.js:446:136916
Failed to publish asset a3f66c60067b06b5d9d00094e9e817ee39dd7cb5c315c8c254f5f3c571959ce5:current_account-current_region
FAILED integ/constructs/xyz/integ.cluster-IntegTest/DefaultTest (undefined/us-east-1) 29.[135](https://github.com/Nextdoor/xxx/actions/runs/6340275359/job/17221306412#step:11:136)s
Integration test failed: TypeError [ERR_STREAM_NULL_VALUES]: May not write null values to stream
```

When we then look in our S3 bucket, we find a series of `22 byte` sized zip files. These three images are from three separate build attempts, all with fresh empty `cdk.out` directories, and all after we had wiped out the S3 cache files:

Screenshot 2023-09-28 at 7 41 49 AM
Screenshot 2023-09-28 at 7 20 22 AM
Screenshot 2023-09-28 at 7 25 17 AM

When we dug into it, it seems that these files are all related to the `onEvent` handlers for the `custom-resource` constructs. Going back in time a bit, it looks like these hash values show up at or around https://github.com/aws/aws-cdk/commit/a9ed64f2aa8014626857dfdfb33a823cd9cfd1fa#diff-8bf3c7acb1f51f01631ea642163612a520b448b843d7514dc31ccc6f140c0753..

## Attempts to fix

### Roll back to `2.90.0` - _success_

We tried to roll back to `2.87.0` - but our codebase would have required too many changes for that, so we were able to roll back to `2.90.0` though which is interestingly before several of the handlers were updated from Node16 to Node18.

When we rolled back to 2.90.0, the integration tests work fine.

### Roll forward to `2.91.0` - _success_

Same as 2.90.0 - the tests work fine.

### Roll forward to `2.92.0` - _partial success_

In https://github.com/aws/aws-cdk/releases/tag/v2.92.0, the custom-resources handler is bumped to use Node18 instead of Node16. That change creates the new asset hash `a3f66c60067b06b5d9d00094e9e817ee39dd7cb5c315c8c254f5f3c571959ce5`. This code mostly worked - however https://github.com/aws/aws-cdk/issues/26771 prevented us from fully testing the CDK construct for EKS.

### Roll forward to `2.93.0` - _success_

In `2.93.0`, we see the asset hash change from `3f579d6c1ab146cac713730c96809dd4a9c5d9750440fb835ab20fd6925e528c.zip -> 9202bb21d52e07810fc1da0f6acf2dcb75a40a43a9a2efbcfc9ae39535c6260c.zip`. It seems that this release works just fine - though the tests are ongoing right now.

### Roll forward to `2.94.0` - _failure_

It seems that the failure starts as soon as we hit the 2.94.0 release.

```
INFRA-MYAPP-ClusterTest: fail: ENOENT: no such file or directory, open '/home/runner/work/infra-myapp/infra-myapp/test/integ/constructs/aws-eks/integ.xx-cluster.ts.snapshot/asset.9202bb21d52e07810fc1da0f6acf2dcb75a40a43a9a2efbcfc9ae39535c6260c.zip'
INFRA-MYAPP-ClusterTest: fail: ENOENT: no such file or directory, open '/home/runner/work/infra-myapp/infra-myapp/test/integ/constructs/aws-eks/integ.xx-cluster.ts.snapshot/asset.e2277687077a2abf9ae1af1cc9565e6715e2ebb62f79ec53aa75a1af9298f642.zip'
INFRA-MYAPP-ClusterTest: fail: ENOENT: no such file or directory, open '/home/runner/work/infra-myapp/inframyapp/test/integ/constructs/aws-eks/integ.xx-cluster.ts.snapshot/asset.8e18eb5caccd2617fb76e648fa6a35dc0ece98c4681942bc6861f41afdff6a1b.zip'
```
### Rolling back to '2.93.9' - _success_

Rolling back to 2.93.0 after the 2.94.0 failure immediately works... builds and integration tests pass again.

### Expected Behavior

A few things here..

1. I obviously don't expect the zip files to be created empty and causing problems.
2. I would expect the files are cleaned up or replaced when they are determined to be corrupt.

### Current Behavior

As far as we can tell, once the corrupt file is created - there are some situations where it is uploaded to S3 (and thus poisoning the cache), and other situations where the upload fails to begin with.

### Reproduction Steps

_Working on this ... don't yet know exactly how to reproduce this_

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### CDK CLI Version

2.95.0+

### Framework Version

_No response_

### Node.js Version

18

### OS

Linux and OSX

### Language

Typescript

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Compare custom-resource handler asset generation between CDK 2.93 and 2.94, beginning with the Node 18 handler change and the reported 22-byte zip files. Use fresh cdk.out directories and cleared S3 caches to investigate; done means corrupt empty zips are not created or uploaded and deployments succeed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, node.js, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.