aws / aws/aws-cdk

(s3-deployment): Support large s3 bucket deployments

Open
#7,950 26 comments 47 reactions 0 assignees View on GitHub
@aws-cdk/aws-s3 @aws-cdk/aws-s3-deployment effort/large feature-request p1
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
1d 19h
Merged PRs (30d)
74

Description

We want to enable users to deploy large amounts of data onto s3 buckets. Currently, such deployments frequently fail because we are limited by the lambda execution environment:

- Deployment size should be no more than 500MB because that is the amount of disk space available on the `/tmp` mount of the lambda.
- `sync` duration has to be less than 15 minutes because this is a hard execution timeout for lambda.

See https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

### Use Case

Users use the `s3-deployment` module to deploy static websites to an s3 bucket. On many occasions, the contents of the website directory can be quite big, and contain many files.

See:

- https://github.com/aws/aws-cdk/issues/7571
- https://github.com/aws/aws-cdk/issues/7316
- https://github.com/aws/aws-cdk/issues/6893

### Proposed Solution

No clear solution to this problem yet, this issue is created so we discuss various approaches.

* [ ] :wave: I may be able to implement this feature request
* [ ] :warning: This feature might incur a breaking change

---

This is a :rocket: Feature Request

Contributor guide

Open the contributing guide

Research direction

The issue names the s3-deployment module but no files, tests, or entry points. Start by locating that module and reviewing how deployments use Lambda, then read the linked issues and AWS Lambda limits. Done requires an agreed approach that supports deployments exceeding the 500 MB and 15-minute constraints, but the issue does not yet define the implementation or validation details.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.