cdk-assets: publish assets in parallel
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the feature
Currently pipelines module uses cdk-assets CLI command to make the publishing of assets to needed accounts. Current approach works fine for Lambdas that go to S3 and for cases where single assets manifest contains only few assets.
There seems to be prop in publishing to publish assets in parallel, but that's not published through CLI.
### Use Case
We're using pipelines module to deploy lambdas (both container and file based) to around 70 target accounts. With just file based lambda's FileAsset's build phase on single job takes around 38 seconds, but at same stage DockerAsset build phase takes around 9 minutes.
If there would be possibility to make image pushes in parallel, it would save quite a lot of our time.
### Proposed Solution
It seems that https://github.com/aws/aws-cdk/blob/5cc0d3514dd6c1bedd8233ec48074257b003fed0/packages/cdk-assets/lib/publishing.ts#L28-L33 has already possibility to parallelize the publishing, but option is not published within CLI at https://github.com/aws/aws-cdk/blob/5cc0d3514dd6c1bedd8233ec48074257b003fed0/packages/cdk-assets/bin/publish.ts#L7-L11.
### Other Information
One possible issue that comes to mind is building, as if publishing is done on parallel (https://github.com/aws/aws-cdk/blob/main/packages/cdk-assets/lib/publishing.ts#L138), also build seems to be included to each one (https://github.com/aws/aws-cdk/blob/main/packages/cdk-assets/lib/publishing.ts#L163-L165). So maybe some refactoring for that part is needed to prevent multiple publishing for making same thing (building) that results same output.
### Acknowledgements
- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
### CDK version used
2.60.0
### Environment details (OS name and version, etc.)
Linux b67dca3f6fa1 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 GNU/Linux
Contributor guide
Research direction
Start with packages/cdk-assets/bin/publish.ts, where the CLI options are defined, and packages/cdk-assets/lib/publishing.ts, where publishing and building are handled. Trace how the CLI invokes publishing and how parallel work affects builds. Done means the CLI exposes parallel asset publishing and avoids repeating equivalent builds when publishing to multiple accounts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cli, cloud, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100