[pipelines] Add stage to strip assets out of cloud assembly before deploying to CloudFormation
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
To avoid the CodePipeline artifact size limit in CloudFormation deploy actions, the pipeline should generate an intermediate artifact which is the cloud assembly but with asset files removed, and use this as the input for the deploy actions.
### Use Case
Regardless of the source provider used, CFN deploy actions have an input artifact size limit of [256MB](https://docs.aws.amazon.com/codepipeline/latest/userguide/limits.html). The CDK pipeline uses the initial cloud assembly, containing all asset files, all the way through to the CFN action inputs, even though the stacks don't require them (as far as I understand the asset system, all assets are published and linked to CFN parameters by this point).
For builds that produce large/multiple assets totalling over 256MB, this causes CodePipeline limit errors in the deployment stages. Assemblies up to 1GB or 5GB (depending on the source provider) could be produced with this change.
Specific example: monorepos used to build many related services that are all deployed as separate containers/functions/etc.
### Proposed Solution
Add an extra pipeline stage after asset publishing and before application stage deployment, which runs a CodeBuild action to load the cloud assembly, strip out asset files, and generate a new artifact containing only the CFN templates and any data necessary for CFN. The CFN actions should use this new artifact as their input.
### Other
* This is currently exaggerated by the lack of a de-dupe option when deploying multiple application stages using the same assets - I believe feature request [#9627](https://github.com/aws/aws-cdk/issues/9627) will reduce code size substantially.
* Overall code size can be reduced by using Lambda layers, but this adds build and deploy complexity compared to using standalone code assets.
* [] :wave: I may be able to implement this feature request
* [] :warning: This feature might incur a breaking change
(:warning: assumes I haven't overlooked some need for the CFN actions to have direct access to asset files)
---
This is a :rocket: Feature Request
Contributor guide
Research direction
Start by tracing the CDK pipeline stages from asset publishing to application stack deployment and identify how CloudFormation deploy actions select their input artifacts. Review the cloud assembly and asset-publishing flow described in the issue, then confirm that a stripped intermediate artifact can retain everything deploy actions need while staying below the artifact limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- ci-cd, cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100