aws / aws/aws-cdk

BucketDeployment: SIGKILL 9 during S3 sync step

Open
#33,512 4 comments 2 reactions 0 assignees View on GitHub
@aws-cdk/aws-s3 bug effort/medium p3
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

After a recent CDK update, I started seeing this error on some deploys when using `BucketDeployment`:

> cfn_error: b"Command '['/opt/awscli/aws', 's3', 'sync', '/tmp/tmps02annce/contents', 's3://beta-us-east-1-main-assets-v1/']' died with ."

The error seems to occur at random, forcing me to frequently retry failed deploys.

I've searched for this same error message but haven't been able to find it. The closest I've found is the issue described at https://github.com/aws/aws-cdk/issues/29862, which is also a SIGKILL but occurs when running the `'aws s3 cp'` command rather than `'aws s3 sync'`. Also, I'm not sure why I'd be running out of memory: The total size of the files I'm deploying is only about 30MB, so I'd expect the 128MB default to be plenty.

I can't say definitively whether this error is caused by a change in CDK or in some other piece of AWS infrastructure.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Last Known Working CDK Version

_No response_

### Expected Behavior

Deploys using `BucketDeployment` should either work or provide a clear error message in the logs when something goes wrong.

### Current Behavior

The error message above is all I see in the logs. It's followed up by a report that says:

> Memory Size: 128 MB Max Memory Used: 128 MB

That suggests that the default memory limit may be the issue… except that I see the same values in the logs when the deploy succeeded. And, again, the files I'm uploading total only 30MB, so what could be using all that memory?

### Reproduction Steps

It's unclear how to reproduce, since the bug is non-deterministic.

### Possible Solution

The improvement proposed at https://github.com/aws/aws-cdk/issues/29898 might address this.

### Additional Information/Context

I'm going to try raising `memoryLimit` to see if it prevents the issue for me. At this point, I don't know.

### CDK CLI Version

2.178.2 (build 89c49cc)

### Framework Version

_No response_

### Node.js Version

20.16.0

### OS

macOS

### Language

Java

### Language Version

Java (17)

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at the BucketDeployment entry point and inspect how the S3 sync step is invoked and how memoryLimit is reported. Compare the SIGKILL behavior with issues 29862 and 29898, then try to reproduce it with a deployment of similar size. Done means identifying the cause and either fixing the failure or providing a clear diagnostic.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.