aws / aws/aws-cdk

[pipelines] Add stage to strip assets out of cloud assembly before deploying to CloudFormation

Open
#9,917 38 comments 84 reactions 0 assignees View on GitHub
@aws-cdk/pipelines effort/medium feature-request p1
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

To avoid the CodePipeline artifact size limit in CloudFormation deploy actions, the pipeline should generate an intermediate artifact which is the cloud assembly but with asset files removed, and use this as the input for the deploy actions.

### Use Case

Regardless of the source provider used, CFN deploy actions have an input artifact size limit of [256MB](https://docs.aws.amazon.com/codepipeline/latest/userguide/limits.html). The CDK pipeline uses the initial cloud assembly, containing all asset files, all the way through to the CFN action inputs, even though the stacks don't require them (as far as I understand the asset system, all assets are published and linked to CFN parameters by this point).

For builds that produce large/multiple assets totalling over 256MB, this causes CodePipeline limit errors in the deployment stages. Assemblies up to 1GB or 5GB (depending on the source provider) could be produced with this change.

Specific example: monorepos used to build many related services that are all deployed as separate containers/functions/etc.

### Proposed Solution

Add an extra pipeline stage after asset publishing and before application stage deployment, which runs a CodeBuild action to load the cloud assembly, strip out asset files, and generate a new artifact containing only the CFN templates and any data necessary for CFN. The CFN actions should use this new artifact as their input.

### Other

* This is currently exaggerated by the lack of a de-dupe option when deploying multiple application stages using the same assets - I believe feature request [#9627](https://github.com/aws/aws-cdk/issues/9627) will reduce code size substantially.
* Overall code size can be reduced by using Lambda layers, but this adds build and deploy complexity compared to using standalone code assets.

* [] :wave: I may be able to implement this feature request
* [] :warning: This feature might incur a breaking change
(:warning: assumes I haven't overlooked some need for the CFN actions to have direct access to asset files)

---

This is a :rocket: Feature Request

Contributor guide

Open the contributing guide

Research direction

Start by tracing the CDK pipeline stages from asset publishing to application stack deployment and identify how CloudFormation deploy actions select their input artifacts. Review the cloud assembly and asset-publishing flow described in the issue, then confirm that a stripped intermediate artifact can retain everything deploy actions need while staying below the artifact limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
ci-cd, cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.