custom-resource-handlers: buffered asset download
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the feature
This issue is part feature request, part bug report.
S3 custom resource should be able to handle arbitrarily large asset files efficiently
### Use Case
Deploying large asset files with an S3 custom resource causes the deployment Lambda to run out of memory The function's memory can be increased to compensate for bigger file sizes, but only up to a point and at additional costs.
### Proposed Solution
It doesn't seem like the AWS CLI `s3` commands support buffered download. As proposed in https://github.com/aws/aws-cdk/issues/29862#issuecomment-2061348326, using boto3's `multipart_threshold` should allow the file to be downloaded and written to disk in multiple parts.
There might be other places where this change would be necessary, but this is the one that caused the initial issue:
https://github.com/aws/aws-cdk/blob/cfd336ee05f4c07c359a4be24011528cb7b55d8f/packages/%40aws-cdk/custom-resource-handlers/lib/aws-s3-deployment/bucket-deployment-handler/index.py#L171-L180
### Other Information
See #29862 for the original issue
### Acknowledgements
- [x] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
### CDK version used
2.134.0
### Environment details (OS name and version, etc.)
N/A
Contributor guide
Research direction
Start at packages/@aws-cdk/custom-resource-handlers/lib/aws-s3-deployment/bucket-deployment-handler/index.py around lines 171-180 and trace how the S3 asset is downloaded. Check whether related download paths also need the same treatment, then verify that arbitrarily large assets can be downloaded without exhausting the deployment Lambda's memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100