aws / aws/aws-cdk

(s3-deployment): Supporting larger artifacts than current file sync buffer allows

Open
#24,569 5 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-s3-deployment effort/medium feature-request p1
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the feature

When trying to use s3-deploy especially in a pipeline, you are restricted to the file sync buffer size (roughly ~2 Gb). It would be nice to be able to configure perhaps a different backend or loading type so that larger artifacts can be uploaded.

I believe because this fails on `deploy` as it tries to read files and send to s3.

Looking at some node documentation, it seems a possible solution (aside from hardware dependent configurations) is that the objects could instead be streamed. Which might not make sense for the synth step, but maybe there is a short circuit that basically holds a pointer to the artifact file and uploads it via streaming method on deploy.

Would love to hear any thoughts and if I've misunderstood the limitations.

### Use Case

This is a pretty big opportunity in the AI/ML space as we see larger and larger models take off, when these models have billions of parameters the compressed weights on disc are in multiple GBs. For commodity level models, being able to support 2-6GB would unlock a lot of potential, as you could use CDK for a full CI/CD pipeline to deploy model and artifacts. For common AWS use cases, breaking apart the weights is not practical.

### Proposed Solution

I'm hoping there is way to alter how the `readFileSync` is used in favor of a way that can chunk or stream artifacts if they are over a certain size.

### Other Information
sample error:
```
[0%] start: Publishing 68b22621fff135f9e3f225bad7ff80fdf2f45c3d9910af601206a0d9b279933a:current_account-current_region
[0%] start: Publishing 2bc265c5e0569aeb24a6349c15bd54e76e845892376515e036627ab0cc70bb64:current_account-current_region
[0%] start: Publishing 8af3d225e619786b1a9f4fbad0864b7414ee4518f4fc009c94e53120126b0002:current_account-current_region
[0%] start: Publishing e44f234dba2a8d3ba8cc58716144cd3ab4a9da9bd72d4e64a19c79cffe22548e:current_account-current_region
node:internal/errors:490
ErrorCaptureStackTrace(err);
^

RangeError [ERR_FS_FILE_TOO_LARGE]: File size (2424025381) is greater than 2 GiB
````

### Acknowledgements

- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### CDK version used

2.17

### Environment details (OS name and version, etc.)

Mac and Linux

Contributor guide

Open the contributing guide

Research direction

Trace the deploy path where artifacts are read with readFileSync and compare it with the reported ERR_FS_FILE_TOO_LARGE failure for a 2.4 GB file. Review how s3-deploy handles artifact uploads, then define and test a streaming or chunked path that successfully publishes artifacts larger than 2 GiB without changing synth behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, node.js, typescript
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.