cdklabs / cdklabs/cdk-stacksets

StackSetStack file assets fail with S3 NoSuchKey when assets are staged with a custom assetHash (aws-cdk-lib >= 2.254 custom-resource handlers)

Open
#895 2 comments 0 reactions 0 assignees View on GitHub
bug effort/medium p1
Dominant language
TypeScript
Stars
118
Forks
25
Avg merge
33m
Merged PRs (30d)
4

Description

## Description

Deploying a `StackSet` whose `StackSetStack` contains an `AwsCustomResource` (or any construct backed by an aws-cdk-lib custom-resource handler) fails on the stack instances with:

```
ResourceLogicalId:AWS679f53fac002430cb0da5b7982bd22872D164C4C, ResourceType:AWS::Lambda::Function,
ResourceStatusReason:Resource handler returned message: "Error occurred while GetObject. S3 Error Code: NoSuchKey.
S3 Error Message: The specified key does not exist. (Service: Lambda, Status Code: 400, ...)"
```

This worked with aws-cdk-lib `2.253.1` and breaks with `2.258.1` (the regression window is somewhere in 2.254–2.258). cdk-stacksets version: `0.0.154`.

## Root cause

`StackSetStackSynthesizer.addFileAsset()` derives the template's S3 object key from the *staged asset file name* ([src/stackset-stack.ts](https://github.com/cdklabs/cdk-stacksets/blob/main/src/stackset-stack.ts)):

```ts
const assetFileBaseName = path.basename(asset.fileName);
const s3Filename = assetFileBaseName.split('.')[1] + '.zip';
const objectKey = `${s3Filename}`;
```

But the object that actually lands in the asset bucket is uploaded by the `BucketDeployment` created in the same method:

```ts
new BucketDeployment(parentStack, bucketDeploymentConstructName, {
sources: [Source.asset(assetPath)],
destinationBucket: assetDeployment.assetBucket,
extract: false,
prune: false,
});
```

`Source.asset(assetPath)` re-stages the already-staged asset directory and computes a **fresh content fingerprint** (`FileSystem.fingerprint`). With `extract: false`, the destination object key is that fingerprint-derived key — not the hash embedded in the staged folder name.

These two keys are only identical when the original asset's hash *is* the plain content fingerprint. That assumption broke: since ~aws-cdk-lib 2.254 the generated custom-resource handler singletons stage their code with a **hardcoded custom `assetHash`**, e.g. in `custom-resource-handlers/dist/custom-resources/aws-custom-resource-provider.generated.js`:

```js
code: lambda.Code.fromAsset(path.join(__dirname, "aws-custom-resource-handler"), {
assetHash: "048c51b339711116c3b9feefa3c6aff37c49dba2a4d0f1f531fa8337d9602d82"
}),
```

So for this asset (concrete values from our deployment):

- staged folder: `cdk.out/asset.8b1c21efe0e24332f92ed71e98107d8a27ee1544c62b7b191805e0913fcfdf0f` (derived from the custom `assetHash`)
- template `Code.S3Key`: `8b1c21efe0e24332f92ed71e98107d8a27ee1544c62b7b191805e0913fcfdf0f.zip`
- key actually uploaded by the BucketDeployment: `0528c0095b5b91654052a1e243a4d0125eb054366baf2bf1c791a871d715313a.zip` (= `FileSystem.fingerprint()` of the staged directory)

The stack instance's Lambda then points at a key that never exists in the asset bucket → `NoSuchKey`.

## Reproduction

```ts
const stackSetStack = new StackSetStack(stack, 'StackSetStack', {
assetBuckets: [bucket],
assetBucketPrefix: 'my-assets',
});

new cr.AwsCustomResource(stackSetStack, 'CustomResource', {
onUpdate: {
service: 'GuardDuty',
action: 'listDetectors',
physicalResourceId: cr.PhysicalResourceId.of('id'),
},
policy: cr.AwsCustomResourcePolicy.fromSdkCalls({ resources: ['*'] }),
});
```

Synth with aws-cdk-lib >= 2.258 and compare the stackset template's `Code.S3Key` with the parent stack's `Custom::CDKBucketDeployment` `SourceObjectKeys` — they differ. Deploying fails with `NoSuchKey` on every stack instance.

## Suggested fix

Derive the template key the same way `Source.asset()` will, so they can never diverge:

```ts
const objectKey = `${FileSystem.fingerprint(assetPath)}.zip`;
```

(Equivalently: pass the original asset hash through to `Source.asset(assetPath, { assetHash: ... })` so the upload lands under the key already written to the template — either direction works as long as both sides agree.)

We've verified the `FileSystem.fingerprint` approach fixes the deployment in our landing zone (all asset keys align again, including plain directory assets and pre-zipped bundled assets, which are unaffected no-ops).

## Environment

- cdk-stacksets: 0.0.154
- aws-cdk-lib: 2.258.1 (works on 2.253.1)
- constructs: 10.6.0
- Node.js: 22

Contributor guide

Open the contributing guide

Research direction

Start in src/stackset-stack.ts and inspect StackSetStackSynthesizer.addFileAsset(), then reproduce the issue with an AwsCustomResource and compare the template Code.S3Key with the parent stack's Custom::CDKBucketDeployment SourceObjectKeys. Done means both keys agree for custom-hashed assets and the StackSet instances deploy without S3 NoSuchKey errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.