aws / aws/aws-cdk

cdk-assets: publish assets in parallel

Open
#23,657 1 comment 2 reactions 0 assignees View on GitHub
@aws-cdk/assets @aws-cdk/core feature-request p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the feature

Currently pipelines module uses cdk-assets CLI command to make the publishing of assets to needed accounts. Current approach works fine for Lambdas that go to S3 and for cases where single assets manifest contains only few assets.

There seems to be prop in publishing to publish assets in parallel, but that's not published through CLI.

### Use Case

We're using pipelines module to deploy lambdas (both container and file based) to around 70 target accounts. With just file based lambda's FileAsset's build phase on single job takes around 38 seconds, but at same stage DockerAsset build phase takes around 9 minutes.

If there would be possibility to make image pushes in parallel, it would save quite a lot of our time.

### Proposed Solution

It seems that https://github.com/aws/aws-cdk/blob/5cc0d3514dd6c1bedd8233ec48074257b003fed0/packages/cdk-assets/lib/publishing.ts#L28-L33 has already possibility to parallelize the publishing, but option is not published within CLI at https://github.com/aws/aws-cdk/blob/5cc0d3514dd6c1bedd8233ec48074257b003fed0/packages/cdk-assets/bin/publish.ts#L7-L11.

### Other Information

One possible issue that comes to mind is building, as if publishing is done on parallel (https://github.com/aws/aws-cdk/blob/main/packages/cdk-assets/lib/publishing.ts#L138), also build seems to be included to each one (https://github.com/aws/aws-cdk/blob/main/packages/cdk-assets/lib/publishing.ts#L163-L165). So maybe some refactoring for that part is needed to prevent multiple publishing for making same thing (building) that results same output.

### Acknowledgements

- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### CDK version used

2.60.0

### Environment details (OS name and version, etc.)

Linux b67dca3f6fa1 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 GNU/Linux

Contributor guide

Open the contributing guide

Research direction

Start with packages/cdk-assets/bin/publish.ts, where the CLI options are defined, and packages/cdk-assets/lib/publishing.ts, where publishing and building are handled. Trace how the CLI invokes publishing and how parallel work affects builds. Done means the CLI exposes parallel asset publishing and avoids repeating equivalent builds when publishing to multiple accounts.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cli, cloud, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.