aws / aws/aws-cdk

(aws-servicecatalog): ProductStack assetBucket in a different stack than ProductStack's parent causes deploy ordering failures

Open
#38,099 1 comment 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-servicecatalog bug effort/medium p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

When a ProductStack uses asset_bucket (e.g. for nested stack templates or Lambda assets) and that bucket is defined in a different CloudFormation stack than the stack that owns the ProductStack, cdk deploy can fail during the ProductAssetsDeployment custom resource (in the bucket stack).

The error returned by CloudFormation is very unintuitive:

> Received response status [FAILED] from custom resource. Message returned: Command '['/opt/awscli/aws', 's3', 'cp', 's3://$CDK_ASSET_BUCKET/$HASH.json', '/tmp/tmp4m6m8mbl/contents']' returned non-zero exit status 1. (RequestId: $request_id)

(For clarity, `$CDK_ASSET_BUCKET` is the general CDK asset bucket, not the ServiceCatalog ProductStack asset bucket.)

Digging into the CloudWatch logs of the associated BucketDeployment Lambda Function gives a little more detail:
> aws s3 cp s3://$CDK_ASSET_BUCKET/$HASH.json /tmp/tmpxvnj9kj2/contents
fatal error: An error occurred (404) when calling the HeadObject operation: Key "$HASH.json" does not exist

---
(Some LLM analysis of the cause - take it with a grain of salt)

The failure happens because:

1. ProductStackSynthesizer.addFileAsset() registers file assets on the ProductStack parent stack’s synthesizer (e.g. StackB.assets.json).
2. ProductAssetsDeployment is created as a child of assetBucket, which lives in a different stack (e.g. StackA).
3. On a multi-stack deploy, StackA may deploy before StackB publishes those assets to the bootstrap bucket.
4. The Custom::CDKBucketDeployment Lambda then calls HeadObject / GetObject on a bootstrap key that does not exist yet → 404 (or similar S3 error).

Co-locating asset_bucket in the same stack as the ProductStack parent avoids the failure.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Last Known Working CDK Library Version

_No response_

### Expected Behavior

Deploying multiple stacks that together define a Service Catalog product with assets should succeed regardless of deploy order, or CDK should document/enforce that assetBucket must live in the same deployable stack as the ProductStack parent.

At minimum, CDK should emit a synth-time warning or add an explicit cross-stack dependency so bootstrap assets are published before ProductAssetsDeployment runs.

### Current Behavior

During cdk deploy of two stacks (bucket stack + product stack), deployment fails on the bucket stack’s ProductAssetsDeployment:

Custom::CDKBucketDeployment ... CREATE_FAILED
Received response status [FAILED] from custom resource. Message returned:
Error: Command '['aws', 's3', 'cp', 's3://cdk-assets--/.json', ...]' returned non-zero exit status 1.
...
An error occurred (404) when calling the HeadObject operation: Not Found

Deploy log pattern:

StackA: check: Check s3://cdk-assets-.../.json ← not found
...
N total assets, M still need to be published ← unpublished assets belong to StackB
StackA | ProductAssetsDeployment/CustomResource/Default ← fails here
StackB | ... publishes assets later (or never in same deploy pass)

In our case:

• StackA (Base): sc_asset_bucket + ProductAssetsDeployment
• StackB (Shared): CloudFormationProduct + OpenSearchParentStack (ProductStack with nested stacks + asset_bucket=self.base.sc_asset_bucket)

Moving sc_asset_bucket into StackB (same stack as the product) fixed the deploy failure.

### Reproduction Steps

Minimal Python repro (two stacks, bucket in A, ProductStack in B):

```
#!/usr/bin/env python3
import aws_cdk as cdk
from aws_cdk import NestedStack, Stack, aws_s3 as s3, aws_servicecatalog as sc
from constructs import Construct

class AssetBucketStack(Stack):
"""Stack A: owns the SC asset bucket only."""
def __init__(self, scope: Construct, id: str, **kwargs):
super().__init__(scope, id, **kwargs)
self.asset_bucket = s3.Bucket(
self,
"ScAssetBucket",
bucket_name=f"sc-asset-demo-{self.account}-{self.region}",
removal_policy=cdk.RemovalPolicy.DESTROY,
auto_delete_objects=True,
)

class ProductNestedStack(NestedStack):
def __init__(self, scope: Construct, id: str):
super().__init__(scope, id)
s3.Bucket(self, "Dummy") # forces nested stack template asset

class DemoProductStack(sc.ProductStack):
def __init__(self, scope: Construct, id: str, *, asset_bucket: s3.IBucket):
super().__init__(scope, id, asset_bucket=asset_bucket)
ProductNestedStack(self, "ProductNestedStack")

class ProductStack(Stack):
"""Stack B: owns the ProductStack + CloudFormationProduct."""
def __init__(self, scope: Construct, id: str, asset_bucket: s3.IBucket, **kwargs):
super().__init__(scope, id, **kwargs)
sc.CloudFormationProduct(
self,
"DemoProduct",
product_name="DemoProduct",
owner="demo",
product_versions=[
sc.CloudFormationProductVersion(
product_version_name="v1",
cloud_formation_template=sc.CloudFormationTemplate.from_product_stack(
DemoProductStack(self, "DemoProductStack", asset_bucket=asset_bucket)
),
)
],
)

app = cdk.App()
bucket_stack = AssetBucketStack(app, "AssetBucketStack")
product_stack = ProductStack(
app,
"ProductStack",
asset_bucket=bucket_stack.asset_bucket, # cross-stack reference
)
# No explicit dependency — CDK may deploy AssetBucketStack first
app.synth()
```

Repro steps:

1. Bootstrap the environment.
2. cdk deploy AssetBucketStack ProductStack (or cdk deploy --all).
3. Observe failure on AssetBucketStack → ScAssetBucket/ProductAssetsDeployment/CustomResource with S3 404 on a bootstrap asset hash registered in
ProductStack.assets.json.

### Possible Solution

Any of:

1. Document that assetBucket must be in the same stack as the ProductStack parent (same as integration tests).
2. Synth-time validation/warning when Stack.of(assetBucket) !== productStack._getParentStack().
3. Fix ordering: add an explicit dependency from the bucket stack’s ProductAssetsDeployment on the parent stack’s asset publishing (or host
ProductAssetsDeployment on the parent stack again, with a stable construct id per bucket).
4. Alternative: register ProductStack file assets on the bucket stack’s synthesizer when the bucket is in a different stack (so publish + copy happen
in one deploy unit).

Relevant implementation: packages/aws-cdk-lib/aws-servicecatalog/lib/private/product-stack-synthesizer.ts — parentStack.synthesizer.addFileAsset() vs
new BucketDeployment(deploymentScope, 'ProductAssetsDeployment', ...) where deploymentScope = this.assetBucket.

### Additional Information/Context

Root cause is a split between where assets are registered (ProductStack parent stack) and where the copy CR lives (asset bucket stack). This follows
from:

• #26311 (https://github.com/aws/aws-cdk/pull/26311) — forward nested-stack assets to parent stack synthesizer, copy from bootstrap via Source.bucket
• #26885 (https://github.com/aws/aws-cdk/pull/26885) — share one ProductAssetsDeployment per bucket, scoped to the bucket construct

Related but not the same issue:

• #24317 (https://github.com/aws/aws-cdk/issues/24317) — synth-time “Cannot find asset” for nested stacks (fixed)
• #25189 (https://github.com/aws/aws-cdk/issues/25189) — duplicate AssetsBucketDeployment construct id (fixed)
• #25733 (https://github.com/aws/aws-cdk/issues/25733) — AwsCliLayer not copied to SC asset bucket

Workaround: define asset_bucket in the same stack as the CloudFormationProduct / ProductStack parent.

### AWS CDK Library version (aws-cdk-lib)

2.1125.0

### AWS CDK CLI version

2.1125.0 (build 71fd29e)

### Node.js Version

v24.16.0

### OS

Fedora 43

### Language

Python

### Language Version

Python 3.11.15

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in packages/aws-cdk-lib/aws-servicecatalog/lib/private/product-stack-synthesizer.ts and trace parentStack.synthesizer.addFileAsset() alongside BucketDeployment with the ProductAssetsDeployment construct. Run the two-stack Python reproduction and inspect the synthesized assets and deployment ordering. Done means the cross-stack case deploys reliably or receives the documented synth-time validation or warning.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.