aws / aws/aws-cdk

(pipelines): Set publishAssetsInParallel to false by default

Open
#23,016 3 comments 5 reactions 0 assignees View on GitHub
@aws-cdk/pipelines effort/small feature-request p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
1d 19h
Merged PRs (30d)
74

Description

### Describe the feature

I initially opened this as a discussion, but it doesn't seem to be getting much visibility there, so I'm opening it as an issue.

Right now, the `publishAssetsInParallel` setting defaults to `true`. While I can appreciate this being useful for some cases, I'd be skeptical of it being a good default setting.

The theoretical advantage is that if you have many assets that need to build and they take a long time to compile, you can parallelize. However, we've found this setting to actually be slower, more expensive, more difficult to debug, and more complicated to maintain.

I'll caveat this by noting that our usage of CDK is around serverless infra that makes primary use of Lambda Node runtimes with TypeScript and local esbuild compilation without Docker.

Regarding cost, I think it's perfectly reasonable to expect that cost would be higher in return for better performance. However, for our usage, we're seeing cost being as much as 20X higher despite being slower. Our build stages for Assets usually complete either under a minute or maybe within a few minutes. But having 20 CodeBuild projects run that all may take under a minute each but collectively get billed for a 1 minute duration minimum each costs way more.

Parallelization should theoretically make things faster, but the overhead of the project setup is actually the bulk of the time for us. Waiting for containers to queue and provision actually often takes minutes on its own. Even if we disregard that, the dependency install is where a lot of time goes and gets duplicated. We found our pipelines to build several minutes faster through the Asset stage with parallelization off.

The burden of CodeBuild project pollution with the randomly generated names from the CDK also makes things a bit of a mess to manage and keep track of. The CodeBuild console is full of noise. There is nothing meaningful about the project for Asset17 failing vs Asset14 at a glance.

### Use Case

Asset builds could be faster, cheaper, and easier to monitor/manage.

Possibly related to #19193. The comments on that PR seem to suggest they found a speedup by making it parallel. I think that speedup holds in local hotswap runs where CodeBuild is bypassed, but it causes a lot of problems in the actual pipeline. Having 50 assets that build in 58 seconds total instead running 50 parallel builds that take 18 seconds results in a 50x cost increase due to the 1 minute minimum billing charge for CodeBuild. Even worse, however, is the slowdown that happens due to delays in CodeBuild container provisioning. We find that parallel builds are an order of magnitude slower than non-parallel builds in practice. It's kind of the worst of both worlds. When you then consider the CodeBuild project pollution that makes the CodeBuild console completely unusable, this really feels like non-parallel builds would be better.

### Proposed Solution

Set `publishAssetsInParallel` to `false` by default. If parallel execution in hotswaps is desired, maybe introduce a separate settings for that or something.

### Other Information

Originally posted discussion:

https://github.com/aws/aws-cdk/discussions/21549

This would be a breaking change in behavior

### Acknowledgements

- [X] I may be able to implement this feature request
- [X] This feature might incur a breaking change

### CDK version used

2.51.1

### Environment details (OS name and version, etc.)

N/A

Contributor guide

Open the contributing guide

Research direction

Start by searching the repository for the `publishAssetsInParallel` setting and its current default, then inspect the pipeline asset publishing entry point and related behavior. Confirm how the default affects regular pipelines and hotswap runs, and verify that the setting defaults to false without changing explicitly configured values.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
build-system, cloud, devops
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.