(custom-resources): empty onEvent handler zip's being created, failing deploys
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
We recently started to see our integration tests failing, even though deploys were succeeding. The failures on the integration tests look like this:
```
sent 1,788 bytes received 35 bytes 3,646.00 bytes/sec
total size is 1,680 speedup is 0.92
fatal: Not a valid object name integ
INFRA-MYAPP-ClusterTest: fail: ENOENT: no such file or directory, open '/home/runner/work/x/xxx/test/integ/constructs/xyz/integ.cluster.ts.snapshot/asset.9202bb21d52e07810fc1da0f6acf2dcb75a40a43a9a2efbcfc9ae39535c6260c.zip'
INFRA-MYAPP-ClusterTest: fail: ENOENT: no such file or directory, open '/home/runner/work/xxx/xxx/test/integ/constructs/xyz/integ.cluster.ts.snapshot/asset.8e18eb5caccd2617fb76e648fa6a35dc0ece98c4681942bc6861f41afdff6a1b.zip'
INFRA-MYAPP-ClusterTest: fail: ENOENT: no such file or directory, open '/home/runner/work/xxx/xxx/test/integ/constructs/xyz/integ.cluster.ts.snapshot/asset.e2277687077a2abf9ae1af1cc9565e6715e2ebb62f79ec53aa75a1af9298f642.zip'
❌ Deployment failed: Error: Failed to publish asset a3f66c60067b06b5d9d00094e9e817ee39dd7cb5c315c8c254f5f3c571959ce5:current_account-current_region
at Deployments.publishSingleAsset (/home/runner/work/xxx/xxx/node_modules/aws-cdk/lib/index.js:446:11458)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Object.publishAsset (/home/runner/work/xxx/xxx/node_modules/aws-cdk/lib/index.js:446:151474)
at async /home/runner/work/xxx/xxx/node_modules/aws-cdk/lib/index.js:446:136916
Failed to publish asset a3f66c60067b06b5d9d00094e9e817ee39dd7cb5c315c8c254f5f3c571959ce5:current_account-current_region
FAILED integ/constructs/xyz/integ.cluster-IntegTest/DefaultTest (undefined/us-east-1) 29.[135](https://github.com/Nextdoor/xxx/actions/runs/6340275359/job/17221306412#step:11:136)s
Integration test failed: TypeError [ERR_STREAM_NULL_VALUES]: May not write null values to stream
```
When we then look in our S3 bucket, we find a series of `22 byte` sized zip files. These three images are from three separate build attempts, all with fresh empty `cdk.out` directories, and all after we had wiped out the S3 cache files:
When we dug into it, it seems that these files are all related to the `onEvent` handlers for the `custom-resource` constructs. Going back in time a bit, it looks like these hash values show up at or around https://github.com/aws/aws-cdk/commit/a9ed64f2aa8014626857dfdfb33a823cd9cfd1fa#diff-8bf3c7acb1f51f01631ea642163612a520b448b843d7514dc31ccc6f140c0753..
## Attempts to fix
### Roll back to `2.90.0` - _success_
We tried to roll back to `2.87.0` - but our codebase would have required too many changes for that, so we were able to roll back to `2.90.0` though which is interestingly before several of the handlers were updated from Node16 to Node18.
When we rolled back to 2.90.0, the integration tests work fine.
### Roll forward to `2.91.0` - _success_
Same as 2.90.0 - the tests work fine.
### Roll forward to `2.92.0` - _partial success_
In https://github.com/aws/aws-cdk/releases/tag/v2.92.0, the custom-resources handler is bumped to use Node18 instead of Node16. That change creates the new asset hash `a3f66c60067b06b5d9d00094e9e817ee39dd7cb5c315c8c254f5f3c571959ce5`. This code mostly worked - however https://github.com/aws/aws-cdk/issues/26771 prevented us from fully testing the CDK construct for EKS.
### Roll forward to `2.93.0` - _success_
In `2.93.0`, we see the asset hash change from `3f579d6c1ab146cac713730c96809dd4a9c5d9750440fb835ab20fd6925e528c.zip -> 9202bb21d52e07810fc1da0f6acf2dcb75a40a43a9a2efbcfc9ae39535c6260c.zip`. It seems that this release works just fine - though the tests are ongoing right now.
### Roll forward to `2.94.0` - _failure_
It seems that the failure starts as soon as we hit the 2.94.0 release.
```
INFRA-MYAPP-ClusterTest: fail: ENOENT: no such file or directory, open '/home/runner/work/infra-myapp/infra-myapp/test/integ/constructs/aws-eks/integ.xx-cluster.ts.snapshot/asset.9202bb21d52e07810fc1da0f6acf2dcb75a40a43a9a2efbcfc9ae39535c6260c.zip'
INFRA-MYAPP-ClusterTest: fail: ENOENT: no such file or directory, open '/home/runner/work/infra-myapp/infra-myapp/test/integ/constructs/aws-eks/integ.xx-cluster.ts.snapshot/asset.e2277687077a2abf9ae1af1cc9565e6715e2ebb62f79ec53aa75a1af9298f642.zip'
INFRA-MYAPP-ClusterTest: fail: ENOENT: no such file or directory, open '/home/runner/work/infra-myapp/inframyapp/test/integ/constructs/aws-eks/integ.xx-cluster.ts.snapshot/asset.8e18eb5caccd2617fb76e648fa6a35dc0ece98c4681942bc6861f41afdff6a1b.zip'
```
### Rolling back to '2.93.9' - _success_
Rolling back to 2.93.0 after the 2.94.0 failure immediately works... builds and integration tests pass again.
### Expected Behavior
A few things here..
1. I obviously don't expect the zip files to be created empty and causing problems.
2. I would expect the files are cleaned up or replaced when they are determined to be corrupt.
### Current Behavior
As far as we can tell, once the corrupt file is created - there are some situations where it is uploaded to S3 (and thus poisoning the cache), and other situations where the upload fails to begin with.
### Reproduction Steps
_Working on this ... don't yet know exactly how to reproduce this_
### Possible Solution
_No response_
### Additional Information/Context
_No response_
### CDK CLI Version
2.95.0+
### Framework Version
_No response_
### Node.js Version
18
### OS
Linux and OSX
### Language
Typescript
### Language Version
_No response_
### Other information
_No response_
Contributor guide
Research direction
Compare custom-resource handler asset generation between CDK 2.93 and 2.94, beginning with the Node 18 handler change and the reported 22-byte zip files. Use fresh cdk.out directories and cleared S3 caches to investigate; done means corrupt empty zips are not created or uploaded and deployments succeed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, node.js, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100