aws / aws/aws-cdk

custom-resource-handlers: buffered asset download

Open
#29,898 1 comment 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-s3 effort/small feature-request p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the feature

This issue is part feature request, part bug report.

S3 custom resource should be able to handle arbitrarily large asset files efficiently

### Use Case

Deploying large asset files with an S3 custom resource causes the deployment Lambda to run out of memory The function's memory can be increased to compensate for bigger file sizes, but only up to a point and at additional costs.

### Proposed Solution

It doesn't seem like the AWS CLI `s3` commands support buffered download. As proposed in https://github.com/aws/aws-cdk/issues/29862#issuecomment-2061348326, using boto3's `multipart_threshold` should allow the file to be downloaded and written to disk in multiple parts.

There might be other places where this change would be necessary, but this is the one that caused the initial issue:
https://github.com/aws/aws-cdk/blob/cfd336ee05f4c07c359a4be24011528cb7b55d8f/packages/%40aws-cdk/custom-resource-handlers/lib/aws-s3-deployment/bucket-deployment-handler/index.py#L171-L180

### Other Information

See #29862 for the original issue

### Acknowledgements

- [x] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### CDK version used

2.134.0

### Environment details (OS name and version, etc.)

N/A

Contributor guide

Open the contributing guide

Research direction

Start at packages/@aws-cdk/custom-resource-handlers/lib/aws-s3-deployment/bucket-deployment-handler/index.py around lines 171-180 and trace how the S3 asset is downloaded. Check whether related download paths also need the same treatment, then verify that arbitrarily large assets can be downloaded without exhausting the deployment Lambda's memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.