aws / aws/aws-cdk

BucketDeployment creates multiple identical, unused AwsCliLayers

Open
#32,907 8 comments 4 reactions 0 assignees View on GitHub
@aws-cdk/aws-s3 bug effort/medium p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

If a stack contains several BucketDeployment constructs, even though the construct will only create one underlying custom resource provider and lambda, it will create a separate AwsCliLayer for each construct. All but one of these lambda layers will be unused.

You can see the bug in the constructor for `BucketDeployment` in `bucket-deployment.ts`. It invokes `new BucketDeploymentSingletonFunction(...)`, which I assume is in charge of not creating multiple identical resource provider lambdas. However, one of the values passed to that function is `layers: [new AwsCliLayer(this, 'AwsCliLayer')]` ([link to code](https://github.com/aws/aws-cdk/blob/fb2b229912f7400f810959eeb51008e680216718/packages/aws-cdk-lib/aws-s3-deployment/lib/bucket-deployment.ts#L354)). This value is evaluated _before_ the singleton function logic is called, which means that a new AwsCliLayer is created unconditionally. Only the first one will actually be referenced by any lambda! This can be confirmed by examining the synthesized template or by looking at the lambda layers in the AWS console.

In addition to being wasteful, this is also causing deployment failures for us, because CFN attempts to create/update all of those layers simultaneously, resulting in a throttling error ("Resource handler returned message: "Rate exceeded (Service: AWSLambdaInternal; Status Code: 400; Error Code: ThrottlingException ..."). This may be the cause of issue #26940, possibly, although their description sounds a little bit different from ours.

What I think should happen instead: Only one AwsCliLayer needs to exist _per stack_. This can be referenced by multiple BucketDeplyments' lambda handlers. Note that even though BucketDeployment may create multiple "singletons" for different configurations (EFS, ephemeral-storage size, etc etc), they can all share the same AwsCliLayer.

Contributor guide

Open the contributing guide

Research direction

Start in packages/aws-cdk-lib/aws-s3-deployment/lib/bucket-deployment.ts at the BucketDeployment constructor and the AwsCliLayer passed to BucketDeploymentSingletonFunction. Synthesize a stack containing several BucketDeployment constructs to verify the current duplicate layers, then trace how the singleton providers share resources. Done means one AwsCliLayer is created per stack and reused by the relevant handlers without changing provider behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.