(aws-servicecatalog): ProductStack assetBucket in a different stack than ProductStack's parent causes deploy ordering failures
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
When a ProductStack uses asset_bucket (e.g. for nested stack templates or Lambda assets) and that bucket is defined in a different CloudFormation stack than the stack that owns the ProductStack, cdk deploy can fail during the ProductAssetsDeployment custom resource (in the bucket stack).
The error returned by CloudFormation is very unintuitive:
> Received response status [FAILED] from custom resource. Message returned: Command '['/opt/awscli/aws', 's3', 'cp', 's3://$CDK_ASSET_BUCKET/$HASH.json', '/tmp/tmp4m6m8mbl/contents']' returned non-zero exit status 1. (RequestId: $request_id)
(For clarity, `$CDK_ASSET_BUCKET` is the general CDK asset bucket, not the ServiceCatalog ProductStack asset bucket.)
Digging into the CloudWatch logs of the associated BucketDeployment Lambda Function gives a little more detail:
> aws s3 cp s3://$CDK_ASSET_BUCKET/$HASH.json /tmp/tmpxvnj9kj2/contents
fatal error: An error occurred (404) when calling the HeadObject operation: Key "$HASH.json" does not exist
---
(Some LLM analysis of the cause - take it with a grain of salt)
The failure happens because:
1. ProductStackSynthesizer.addFileAsset() registers file assets on the ProductStack parent stack’s synthesizer (e.g. StackB.assets.json).
2. ProductAssetsDeployment is created as a child of assetBucket, which lives in a different stack (e.g. StackA).
3. On a multi-stack deploy, StackA may deploy before StackB publishes those assets to the bootstrap bucket.
4. The Custom::CDKBucketDeployment Lambda then calls HeadObject / GetObject on a bootstrap key that does not exist yet → 404 (or similar S3 error).
Co-locating asset_bucket in the same stack as the ProductStack parent avoids the failure.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Last Known Working CDK Library Version
_No response_
### Expected Behavior
Deploying multiple stacks that together define a Service Catalog product with assets should succeed regardless of deploy order, or CDK should document/enforce that assetBucket must live in the same deployable stack as the ProductStack parent.
At minimum, CDK should emit a synth-time warning or add an explicit cross-stack dependency so bootstrap assets are published before ProductAssetsDeployment runs.
### Current Behavior
During cdk deploy of two stacks (bucket stack + product stack), deployment fails on the bucket stack’s ProductAssetsDeployment:
Custom::CDKBucketDeployment ... CREATE_FAILED
Received response status [FAILED] from custom resource. Message returned:
Error: Command '['aws', 's3', 'cp', 's3://cdk-assets--/.json', ...]' returned non-zero exit status 1.
...
An error occurred (404) when calling the HeadObject operation: Not Found
Deploy log pattern:
StackA: check: Check s3://cdk-assets-.../.json ← not found
...
N total assets, M still need to be published ← unpublished assets belong to StackB
StackA | ProductAssetsDeployment/CustomResource/Default ← fails here
StackB | ... publishes assets later (or never in same deploy pass)
In our case:
• StackA (Base): sc_asset_bucket + ProductAssetsDeployment
• StackB (Shared): CloudFormationProduct + OpenSearchParentStack (ProductStack with nested stacks + asset_bucket=self.base.sc_asset_bucket)
Moving sc_asset_bucket into StackB (same stack as the product) fixed the deploy failure.
### Reproduction Steps
Minimal Python repro (two stacks, bucket in A, ProductStack in B):
```
#!/usr/bin/env python3
import aws_cdk as cdk
from aws_cdk import NestedStack, Stack, aws_s3 as s3, aws_servicecatalog as sc
from constructs import Construct
class AssetBucketStack(Stack):
"""Stack A: owns the SC asset bucket only."""
def __init__(self, scope: Construct, id: str, **kwargs):
super().__init__(scope, id, **kwargs)
self.asset_bucket = s3.Bucket(
self,
"ScAssetBucket",
bucket_name=f"sc-asset-demo-{self.account}-{self.region}",
removal_policy=cdk.RemovalPolicy.DESTROY,
auto_delete_objects=True,
)
class ProductNestedStack(NestedStack):
def __init__(self, scope: Construct, id: str):
super().__init__(scope, id)
s3.Bucket(self, "Dummy") # forces nested stack template asset
class DemoProductStack(sc.ProductStack):
def __init__(self, scope: Construct, id: str, *, asset_bucket: s3.IBucket):
super().__init__(scope, id, asset_bucket=asset_bucket)
ProductNestedStack(self, "ProductNestedStack")
class ProductStack(Stack):
"""Stack B: owns the ProductStack + CloudFormationProduct."""
def __init__(self, scope: Construct, id: str, asset_bucket: s3.IBucket, **kwargs):
super().__init__(scope, id, **kwargs)
sc.CloudFormationProduct(
self,
"DemoProduct",
product_name="DemoProduct",
owner="demo",
product_versions=[
sc.CloudFormationProductVersion(
product_version_name="v1",
cloud_formation_template=sc.CloudFormationTemplate.from_product_stack(
DemoProductStack(self, "DemoProductStack", asset_bucket=asset_bucket)
),
)
],
)
app = cdk.App()
bucket_stack = AssetBucketStack(app, "AssetBucketStack")
product_stack = ProductStack(
app,
"ProductStack",
asset_bucket=bucket_stack.asset_bucket, # cross-stack reference
)
# No explicit dependency — CDK may deploy AssetBucketStack first
app.synth()
```
Repro steps:
1. Bootstrap the environment.
2. cdk deploy AssetBucketStack ProductStack (or cdk deploy --all).
3. Observe failure on AssetBucketStack → ScAssetBucket/ProductAssetsDeployment/CustomResource with S3 404 on a bootstrap asset hash registered in
ProductStack.assets.json.
### Possible Solution
Any of:
1. Document that assetBucket must be in the same stack as the ProductStack parent (same as integration tests).
2. Synth-time validation/warning when Stack.of(assetBucket) !== productStack._getParentStack().
3. Fix ordering: add an explicit dependency from the bucket stack’s ProductAssetsDeployment on the parent stack’s asset publishing (or host
ProductAssetsDeployment on the parent stack again, with a stable construct id per bucket).
4. Alternative: register ProductStack file assets on the bucket stack’s synthesizer when the bucket is in a different stack (so publish + copy happen
in one deploy unit).
Relevant implementation: packages/aws-cdk-lib/aws-servicecatalog/lib/private/product-stack-synthesizer.ts — parentStack.synthesizer.addFileAsset() vs
new BucketDeployment(deploymentScope, 'ProductAssetsDeployment', ...) where deploymentScope = this.assetBucket.
### Additional Information/Context
Root cause is a split between where assets are registered (ProductStack parent stack) and where the copy CR lives (asset bucket stack). This follows
from:
• #26311 (https://github.com/aws/aws-cdk/pull/26311) — forward nested-stack assets to parent stack synthesizer, copy from bootstrap via Source.bucket
• #26885 (https://github.com/aws/aws-cdk/pull/26885) — share one ProductAssetsDeployment per bucket, scoped to the bucket construct
Related but not the same issue:
• #24317 (https://github.com/aws/aws-cdk/issues/24317) — synth-time “Cannot find asset” for nested stacks (fixed)
• #25189 (https://github.com/aws/aws-cdk/issues/25189) — duplicate AssetsBucketDeployment construct id (fixed)
• #25733 (https://github.com/aws/aws-cdk/issues/25733) — AwsCliLayer not copied to SC asset bucket
Workaround: define asset_bucket in the same stack as the CloudFormationProduct / ProductStack parent.
### AWS CDK Library version (aws-cdk-lib)
2.1125.0
### AWS CDK CLI version
2.1125.0 (build 71fd29e)
### Node.js Version
v24.16.0
### OS
Fedora 43
### Language
Python
### Language Version
Python 3.11.15
### Other information
_No response_
Contributor guide
Research direction
Start in packages/aws-cdk-lib/aws-servicecatalog/lib/private/product-stack-synthesizer.ts and trace parentStack.synthesizer.addFileAsset() alongside BucketDeployment with the ProductAssetsDeployment construct. Run the two-stack Python reproduction and inspect the synthesized assets and deployment ordering. Done means the cross-stack case deploys reliably or receives the documented synth-time validation or warning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100