aws / aws/aws-cdk

(codepipeline): Cross-region support stacks break when their template exceeds 50kb

Open
#27,196 1 comment 1 reaction 0 assignees View on GitHub
@aws-cdk/aws-codepipeline bug effort/medium p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
1d 19h
Merged PRs (30d)
74

Description

### Describe the bug

Currently, cross-region support stacks [use the `BootstraplessSynthesizer` by default](https://github.com/aws/aws-cdk/blob/d9f1f0908a1f7ec0bfcb0df4f9dd1b45c86b0144/packages/aws-cdk-lib/aws-codepipeline/lib/pipeline.ts#L681:L689), reusing the pipeline environment's default execution and deploy roles and thus avoiding the requirement to bootstrap all regions you want your actions to use.

This works fine most of the time, until their template reaches the CloudFormation size limit for including the template in a request, which is [`51,200 bytes`](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-limits.html). When this happens, the template needs to be uploaded to S3 as an asset instead - and this fails.

We have seen this with CDK Pipelines, and what happens in practice is that the `SelfMutation` Action tries to upload the template to the default asset bucket in the target region, but fails to do so (as it should, because it lacks the access to do that).

### Expected Behavior

Pipeline works fine.

### Current Behavior

Pipeline tries and fails to upload the support stack template to the asset bucket in the target region.

### Reproduction Steps

Deploy a CDK Pipeline that deploys to a lot of accounts in another region such that the support stack reaches 50kb.

### Possible Solution

Generate a descriptive error when a cross-region support stack reaches 50Kb. Add a flag that would allow us to control the synthesizer to be used in support stacks (or just an option to use the parent one).

I would be glad to open a PR, but would need some guidance on how to detect the case.

### Additional Information/Context

A very hacky workaround is to add explicit statements allowing S3 access to the asset bucket in the target region to the SelfMutation Role, as well as the CDKToolkit's deploy role in the local region. The target region must be bootstrapped.

### CDK CLI Version

2.94.0

### Framework Version

2.94.0

### Node.js Version

18.4.1

### OS

MacOS

### Language

Typescript, Python, .NET, Java, Go

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in packages/aws-cdk-lib/aws-codepipeline/lib/pipeline.ts around the BootstraplessSynthesizer setup at lines 681-689, then trace how the SelfMutation action handles oversized cross-region support stack templates. Reproduce the failure with a support stack exceeding 50 KB and determine whether the intended completion is a descriptive error, configurable synthesizer selection, or parent synthesizer reuse.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.