aws / aws/aws-cdk

Lambda: cdk deploy suddenly getting ECONNRESET on asset publish

Open
#28,763 3 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-lambda @aws-cdk/aws-s3 bug effort/medium p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

I have a stack that contains docker images. I've been deploying it locally for the past 4 months without a hitch (deploy also worked on my partner's computer), and then suddenly today I've started getting `fail: read ECONNRESET`. I'm using `PythonLambda` construct, as well as `DockerImageFunction` construct. All fail.

When running `cdk deploy` with `--verbose` I see these errors are as follows:
```
[23:47:53] Call failed: getBucketLocation({"Bucket":"cdk-hnb659fds-assets-REDACTED-us-east-1"}) => read ECONNRESET (code=TimeoutError)
```
and
```
[23:47:32] Call failed: listObjectsV2({"Bucket":"cdk-hnb659fds-assets-REDACTED-us-east-1","Prefix":"1ce78b814fad9fc710c056afbe94fd1fe40b81d185a68ffa975c6079c9a9cd53.zip","MaxKeys":1}) => read ECONNRESET (code=TimeoutError)
```

Other AWS CLI requests to the account work (such as to `ECR`).

This started today on both computers I and my partner have access to. Could not find a solution for the life of me. both AWS CLI and AWS CDK, and all libraries are on latest versions.

Deploy finishes with:
```
Deployment failed: Error: Failed to publish asset 109bd108d6c60e825885f808d1cf43f42060382afaceed0935d0028c2d821a18:REDACTED-us-east-1
at Deployments.publishSingleAsset (/Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:11655)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Object.publishAsset (/Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:180233)
at async /Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:164772
```

No issues on AWS status pages reported for S3.

### Expected Behavior

Stack should deploy and assets should be published.

### Current Behavior

Failing with:
```
Deployment failed: Error: Failed to publish asset 109bd108d6c60e825885f808d1cf43f42060382afaceed0935d0028c2d821a18:REDACTED-us-east-1
at Deployments.publishSingleAsset (/Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:11655)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Object.publishAsset (/Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:180233)
at async /Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:164772
```

after getting:
```
[23:47:53] Call failed: getBucketLocation({"Bucket":"cdk-hnb659fds-assets-REDACTED-us-east-1"}) => read ECONNRESET (code=TimeoutError)
```
and
```
[23:47:32] Call failed: listObjectsV2({"Bucket":"cdk-hnb659fds-assets-REDACTED-us-east-1","Prefix":"1ce78b814fad9fc710c056afbe94fd1fe40b81d185a68ffa975c6079c9a9cd53.zip","MaxKeys":1}) => read ECONNRESET (code=TimeoutError)
```

### Reproduction Steps

Create a stack in `us-east-1` which uses the `PythonLambda` or the `DockerImageFunction` constructs. Example usage:
```ts
const lambda = new lambda.DockerImageFunction(this, 'some-name', {
functionName: 'some-name',
code: lambda.DockerImageCode.fromImageAsset('path/to/dockerfile'),
memorySize: 2048,
ephemeralStorageSize: cdk.Size.gibibytes(2),
architecture: lambda.Architecture.X86_64,
timeout: cdk.Duration.minutes(10),
retryAttempts: 0,
})
```

### Possible Solution

I have read somewhere that this is because the underlying AWS SDK uses V4 signatures, and S3 expects V2. Not sure if that is the case here, or why it would change all of a sudden.

### Additional Information/Context

I have already tried several versions of `CDK`, as well as to delete the `CDKToolkit` stack and reinstall it. Nothing worked.

### CDK CLI Version

2.122.0 (build 7e77e02)

### Framework Version

_No response_

### Node.js Version

v18.9.0

### OS

MacOS 14.2.1 (23C71) (Sonoma)

### Language

TypeScript

### Language Version

_No response_

### Other information

Thank you <3

Contributor guide

Open the contributing guide

Research direction

Start with the `Deployments.publishSingleAsset` entry point and the related code in `aws-cdk/lib/index.js`, using the verbose `getBucketLocation` and `listObjectsV2` ECONNRESET calls as the reproduction. Trace asset publishing for the S3 bootstrap bucket and verify that `cdk deploy` can publish the asset successfully without the timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.