Lambda: cdk deploy suddenly getting ECONNRESET on asset publish
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
I have a stack that contains docker images. I've been deploying it locally for the past 4 months without a hitch (deploy also worked on my partner's computer), and then suddenly today I've started getting `fail: read ECONNRESET`. I'm using `PythonLambda` construct, as well as `DockerImageFunction` construct. All fail.
When running `cdk deploy` with `--verbose` I see these errors are as follows:
```
[23:47:53] Call failed: getBucketLocation({"Bucket":"cdk-hnb659fds-assets-REDACTED-us-east-1"}) => read ECONNRESET (code=TimeoutError)
```
and
```
[23:47:32] Call failed: listObjectsV2({"Bucket":"cdk-hnb659fds-assets-REDACTED-us-east-1","Prefix":"1ce78b814fad9fc710c056afbe94fd1fe40b81d185a68ffa975c6079c9a9cd53.zip","MaxKeys":1}) => read ECONNRESET (code=TimeoutError)
```
Other AWS CLI requests to the account work (such as to `ECR`).
This started today on both computers I and my partner have access to. Could not find a solution for the life of me. both AWS CLI and AWS CDK, and all libraries are on latest versions.
Deploy finishes with:
```
Deployment failed: Error: Failed to publish asset 109bd108d6c60e825885f808d1cf43f42060382afaceed0935d0028c2d821a18:REDACTED-us-east-1
at Deployments.publishSingleAsset (/Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:11655)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Object.publishAsset (/Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:180233)
at async /Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:164772
```
No issues on AWS status pages reported for S3.
### Expected Behavior
Stack should deploy and assets should be published.
### Current Behavior
Failing with:
```
Deployment failed: Error: Failed to publish asset 109bd108d6c60e825885f808d1cf43f42060382afaceed0935d0028c2d821a18:REDACTED-us-east-1
at Deployments.publishSingleAsset (/Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:11655)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Object.publishAsset (/Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:180233)
at async /Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:164772
```
after getting:
```
[23:47:53] Call failed: getBucketLocation({"Bucket":"cdk-hnb659fds-assets-REDACTED-us-east-1"}) => read ECONNRESET (code=TimeoutError)
```
and
```
[23:47:32] Call failed: listObjectsV2({"Bucket":"cdk-hnb659fds-assets-REDACTED-us-east-1","Prefix":"1ce78b814fad9fc710c056afbe94fd1fe40b81d185a68ffa975c6079c9a9cd53.zip","MaxKeys":1}) => read ECONNRESET (code=TimeoutError)
```
### Reproduction Steps
Create a stack in `us-east-1` which uses the `PythonLambda` or the `DockerImageFunction` constructs. Example usage:
```ts
const lambda = new lambda.DockerImageFunction(this, 'some-name', {
functionName: 'some-name',
code: lambda.DockerImageCode.fromImageAsset('path/to/dockerfile'),
memorySize: 2048,
ephemeralStorageSize: cdk.Size.gibibytes(2),
architecture: lambda.Architecture.X86_64,
timeout: cdk.Duration.minutes(10),
retryAttempts: 0,
})
```
### Possible Solution
I have read somewhere that this is because the underlying AWS SDK uses V4 signatures, and S3 expects V2. Not sure if that is the case here, or why it would change all of a sudden.
### Additional Information/Context
I have already tried several versions of `CDK`, as well as to delete the `CDKToolkit` stack and reinstall it. Nothing worked.
### CDK CLI Version
2.122.0 (build 7e77e02)
### Framework Version
_No response_
### Node.js Version
v18.9.0
### OS
MacOS 14.2.1 (23C71) (Sonoma)
### Language
TypeScript
### Language Version
_No response_
### Other information
Thank you <3
Contributor guide
Research direction
Start with the `Deployments.publishSingleAsset` entry point and the related code in `aws-cdk/lib/index.js`, using the verbose `getBucketLocation` and `listObjectsV2` ECONNRESET calls as the reproduction. Trace asset publishing for the S3 bootstrap bucket and verify that `cdk deploy` can publish the asset successfully without the timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100