(s3): (CustomCDKBucketDeployment custom resource Lambda function fails when updating Bucket `id`)
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
I have a stack using a `BucketDeployment` resource configured with `retainOnDelete: false` using an S3 Bucket configured with `autoDeleteObjects: true`.
When updating the `s3.Bucket` resource `id` and redeploying, the stack fails with the following error:
```
10:10:46 AM | UPDATE_FAILED | Custom::CDKBucketDeployment | stack1CustomResource4003511E
Received response status [FAILED] from custom resource. Message returned: Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deployst
ack-bucket43879c71-1hhkjuylg244r/', '--recursive']' returned non-zero exit status 1. (RequestId: 65867ce6-f4d3-4c8e-bf8e-46dacee629
54)
```
### Expected Behavior
No errors are thrown
### Current Behavior
The following error is thrown:
```
10:10:46 AM | UPDATE_FAILED | Custom::CDKBucketDeployment | stack1CustomResource4003511E
Received response status [FAILED] from custom resource. Message returned: Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deployst
ack-bucket43879c71-1hhkjuylg244r/', '--recursive']' returned non-zero exit status 1. (RequestId: 65867ce6-f4d3-4c8e-bf8e-46dacee629
54)
```
The logs for the CDKBucketDeployment Lambda functions are as follow:
```
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| timestamp | message |
|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1698137767364 | INIT_START Runtime Version: python:3.9.v34 Runtime Version ARN: arn:aws:lambda:eu-west-1::runtime:25730c9630b4654f0753a405e39f452e48dad54fbcae4ce598d051c13d109bf8 |
| 1698137767639 | [INFO] 2023-10-24T08:56:07.639Z Found credentials in environment variables. |
| 1698137767798 | START RequestId: 7fed9806-b5ad-438c-8f64-4eab012fa671 Version: $LATEST |
| 1698137767798 | [INFO] 2023-10-24T08:56:07.798Z 7fed9806-b5ad-438c-8f64-4eab012fa671 {'RequestType': 'Create', 'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'StackId': 'arn:aws:cloudformation:eu-west-1:699391019698:stack/S3DeployStack/0288af90-724b-11ee-af8f-0673370ccdbf', 'RequestId': '0e3b941c-7fbf-4e0c-94ae-92b06815383d', 'LogicalResourceId': 'stack1CustomResource4003511E', 'ResourceType': 'Custom::CDKBucketDeployment', 'ResourceProperties': {'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'RetainOnDelete': 'false', 'Prune': 'true', 'SourceBucketNames': ['cdk-hnb659fds-assets-699391019698-eu-west-1'], 'DestinationBucketName': 's3deploystack-bucket43879c71-efwtvnkgutvp', 'SourceObjectKeys': ['8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip']}} |
| 1698137767799 | [INFO] 2023-10-24T08:56:07.799Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | s3_dest: s3://s3deploystack-bucket43879c71-efwtvnkgutvp/ |
| 1698137767799 | [INFO] 2023-10-24T08:56:07.799Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | old_s3_dest: None |
| 1698137767799 | [INFO] 2023-10-24T08:56:07.799Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | workdir: /tmp/tmpfsiuz_zo |
| 1698137767817 | [INFO] 2023-10-24T08:56:07.816Z 7fed9806-b5ad-438c-8f64-4eab012fa671 archive: /tmp/tmpfsiuz_zo/b8c519a1-28bd-4546-912e-43a04d153982 |
| 1698137767817 | [INFO] 2023-10-24T08:56:07.817Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | aws s3 cp s3://cdk-hnb659fds-assets-699391019698-eu-west-1/8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip /tmp/tmpfsiuz_zo/b8c519a1-28bd-4546-912e-43a04d153982 |
| 1698137787617 | Completed 2.7 KiB/2.7 KiB (9.8 KiB/s) with 1 file(s) remaining download: s3://cdk-hnb659fds-assets-699391019698-eu-west-1/8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip to ../../tmp/tmpfsiuz_zo/b8c519a1-28bd-4546-912e-43a04d153982 |
| 1698137789217 | [INFO] 2023-10-24T08:56:29.217Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | extracting archive to: /tmp/tmpfsiuz_zo/contents |
| 1698137789217 | [INFO] 2023-10-24T08:56:29.217Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | markers: {} |
| 1698137789238 | [INFO] 2023-10-24T08:56:29.238Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | aws s3 sync --delete /tmp/tmpfsiuz_zo/contents s3://s3deploystack-bucket43879c71-efwtvnkgutvp/ |
| 1698137806037 | Completed 2.8 KiB/3.8 KiB (17.3 KiB/s) with 3 file(s) remaining upload: ../../tmp/tmpfsiuz_zo/contents/s3-deploy.js to s3://s3deploystack-bucket43879c71-efwtvnkgutvp/s3-deploy.js |
| 1698137806377 | Completed 2.8 KiB/3.8 KiB (17.3 KiB/s) with 2 file(s) remaining Completed 2.8 KiB/3.8 KiB (5.5 KiB/s) with 2 file(s) remaining upload: ../../tmp/tmpfsiuz_zo/contents/s3-deploy.d.ts to s3://s3deploystack-bucket43879c71-efwtvnkgutvp/s3-deploy.d.ts |
| 1698137806417 | Completed 2.8 KiB/3.8 KiB (5.5 KiB/s) with 1 file(s) remaining Completed 3.8 KiB/3.8 KiB (6.8 KiB/s) with 1 file(s) remaining upload: ../../tmp/tmpfsiuz_zo/contents/s3-deploy.ts to s3://s3deploystack-bucket43879c71-efwtvnkgutvp/s3-deploy.ts |
| 1698137807980 | [INFO] 2023-10-24T08:56:47.980Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | response body: {"Status": "SUCCESS", "Reason": "See the details in CloudWatch Log Stream: 2023/10/24/[$LATEST]0ed22de7de3b4371b62cb69ec2abf684", "PhysicalResourceId": "aws.cdk.s3deployment.a6226748-00ba-435a-a397-1930e4c610a0", "StackId": "arn:aws:cloudformation:eu-west-1:699391019698:stack/S3DeployStack/0288af90-724b-11ee-af8f-0673370ccdbf", "RequestId": "0e3b941c-7fbf-4e0c-94ae-92b06815383d", "LogicalResourceId": "stack1CustomResource4003511E", "NoEcho": false, "Data": {"DestinationBucketArn": null, "SourceObjectKeys": ["8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip"]}} |
| 1698137808352 | [INFO] 2023-10-24T08:56:48.352Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | status code: OK |
| 1698137808359 | END RequestId: 7fed9806-b5ad-438c-8f64-4eab012fa671 |
| 1698137808359 | REPORT RequestId: 7fed9806-b5ad-438c-8f64-4eab012fa671 Duration: 40560.05 ms Billed Duration: 40561 ms Memory Size: 128 MB Max Memory Used: 128 MB Init Duration: 432.95 ms |
| 1698138013699 | START RequestId: bb8ea787-c893-4e8e-ad49-6dfc128d4d6f Version: $LATEST |
| 1698138013700 | [INFO] 2023-10-24T09:00:13.700Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f {'RequestType': 'Update', 'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'StackId': 'arn:aws:cloudformation:eu-west-1:699391019698:stack/S3DeployStack/0288af90-724b-11ee-af8f-0673370ccdbf', 'RequestId': 'f999927a-bc46-4be2-aebf-4897423dd642', 'LogicalResourceId': 'stack1CustomResource4003511E', 'PhysicalResourceId': 'aws.cdk.s3deployment.a6226748-00ba-435a-a397-1930e4c610a0', 'ResourceType': 'Custom::CDKBucketDeployment', 'ResourceProperties': {'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'RetainOnDelete': 'false', 'Prune': 'true', 'SourceBucketNames': ['cdk-hnb659fds-assets-699391019698-eu-west-1'], 'DestinationBucketName': 's3deploystack-bucket136091df3-fmpx07qm4scw', 'SourceObjectKeys': ['8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip']}, 'OldResourceProperties': {'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'RetainOnDelete': 'false', 'Prune': 'true', 'SourceBucketNames': ['cdk-hnb659fds-assets-699391019698-eu-west-1'], 'DestinationBucketName': 's3deploystack-bucket43879c71-efwtvnkgutvp', 'SourceObjectKeys': ['8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip']}} |
| 1698138013700 | [INFO] 2023-10-24T09:00:13.700Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f | s3_dest: s3://s3deploystack-bucket136091df3-fmpx07qm4scw/ |
| 1698138013700 | [INFO] 2023-10-24T09:00:13.700Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f | old_s3_dest: s3://s3deploystack-bucket43879c71-efwtvnkgutvp/ |
| 1698138013700 | [INFO] 2023-10-24T09:00:13.700Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f | aws s3 rm s3://s3deploystack-bucket43879c71-efwtvnkgutvp/ --recursive |
| 1698138030279 | fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied |
| 1698138031879 | [ERROR] 2023-10-24T09:00:31.859Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket43879c71-efwtvnkgutvp/', '--recursive']' returned non-zero exit status 1. Traceback (most recent call last): File "/var/task/index.py", line 115, in handler aws_command("s3", "rm", old_s3_dest, "--recursive") File "/var/task/index.py", line 233, in aws_command subprocess.check_call([aws] + list(args)) File "/var/lang/lib/python3.9/subprocess.py", line 373, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket43879c71-efwtvnkgutvp/', '--recursive']' returned non-zero exit status 1. |
| 1698138031879 | [ERROR] 2023-10-24T09:00:31.879Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f | cfn_error: Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket43879c71-efwtvnkgutvp/', '--recursive']' returned non-zero exit status 1. |
| 1698138031880 | [INFO] 2023-10-24T09:00:31.880Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f | response body: {"Status": "FAILED", "Reason": "Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket43879c71-efwtvnkgutvp/', '--recursive']' returned non-zero exit status 1.", "PhysicalResourceId": "aws.cdk.s3deployment.a6226748-00ba-435a-a397-1930e4c610a0", "StackId": "arn:aws:cloudformation:eu-west-1:699391019698:stack/S3DeployStack/0288af90-724b-11ee-af8f-0673370ccdbf", "RequestId": "f999927a-bc46-4be2-aebf-4897423dd642", "LogicalResourceId": "stack1CustomResource4003511E", "NoEcho": false, "Data": {}} |
| 1698138032124 | [INFO] 2023-10-24T09:00:32.124Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f | status code: OK |
| 1698138032159 | END RequestId: bb8ea787-c893-4e8e-ad49-6dfc128d4d6f |
| 1698138032159 | REPORT RequestId: bb8ea787-c893-4e8e-ad49-6dfc128d4d6f Duration: 18459.66 ms Billed Duration: 18460 ms Memory Size: 128 MB Max Memory Used: 128 MB |
| 1698138055127 | START RequestId: b698df79-1457-4fdf-9c1e-5555b327c105 Version: $LATEST |
| 1698138055127 | [INFO] 2023-10-24T09:00:55.127Z b698df79-1457-4fdf-9c1e-5555b327c105 {'RequestType': 'Update', 'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'StackId': 'arn:aws:cloudformation:eu-west-1:699391019698:stack/S3DeployStack/0288af90-724b-11ee-af8f-0673370ccdbf', 'RequestId': '85ff64d4-c59b-425b-963e-8d85ca9e9191', 'LogicalResourceId': 'stack1CustomResource4003511E', 'PhysicalResourceId': 'aws.cdk.s3deployment.a6226748-00ba-435a-a397-1930e4c610a0', 'ResourceType': 'Custom::CDKBucketDeployment', 'ResourceProperties': {'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'RetainOnDelete': 'false', 'Prune': 'true', 'SourceBucketNames': ['cdk-hnb659fds-assets-699391019698-eu-west-1'], 'DestinationBucketName': 's3deploystack-bucket43879c71-efwtvnkgutvp', 'SourceObjectKeys': ['8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip']}, 'OldResourceProperties': {'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'RetainOnDelete': 'false', 'Prune': 'true', 'SourceBucketNames': ['cdk-hnb659fds-assets-699391019698-eu-west-1'], 'DestinationBucketName': 's3deploystack-bucket136091df3-fmpx07qm4scw', 'SourceObjectKeys': ['8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip']}} |
| 1698138055128 | [INFO] 2023-10-24T09:00:55.128Z b698df79-1457-4fdf-9c1e-5555b327c105 | s3_dest: s3://s3deploystack-bucket43879c71-efwtvnkgutvp/ |
| 1698138055128 | [INFO] 2023-10-24T09:00:55.128Z b698df79-1457-4fdf-9c1e-5555b327c105 | old_s3_dest: s3://s3deploystack-bucket136091df3-fmpx07qm4scw/ |
| 1698138055128 | [INFO] 2023-10-24T09:00:55.128Z b698df79-1457-4fdf-9c1e-5555b327c105 | aws s3 rm s3://s3deploystack-bucket136091df3-fmpx07qm4scw/ --recursive |
| 1698138071759 | fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied |
| 1698138073399 | [ERROR] 2023-10-24T09:01:13.399Z b698df79-1457-4fdf-9c1e-5555b327c105 Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket136091df3-fmpx07qm4scw/', '--recursive']' returned non-zero exit status 1. Traceback (most recent call last): File "/var/task/index.py", line 115, in handler aws_command("s3", "rm", old_s3_dest, "--recursive") File "/var/task/index.py", line 233, in aws_command subprocess.check_call([aws] + list(args)) File "/var/lang/lib/python3.9/subprocess.py", line 373, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket136091df3-fmpx07qm4scw/', '--recursive']' returned non-zero exit status 1. |
| 1698138073399 | [ERROR] 2023-10-24T09:01:13.399Z b698df79-1457-4fdf-9c1e-5555b327c105 | cfn_error: Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket136091df3-fmpx07qm4scw/', '--recursive']' returned non-zero exit status 1. |
| 1698138073400 | [INFO] 2023-10-24T09:01:13.400Z b698df79-1457-4fdf-9c1e-5555b327c105 | response body: {"Status": "FAILED", "Reason": "Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket136091df3-fmpx07qm4scw/', '--recursive']' returned non-zero exit status 1.", "PhysicalResourceId": "aws.cdk.s3deployment.a6226748-00ba-435a-a397-1930e4c610a0", "StackId": "arn:aws:cloudformation:eu-west-1:699391019698:stack/S3DeployStack/0288af90-724b-11ee-af8f-0673370ccdbf", "RequestId": "85ff64d4-c59b-425b-963e-8d85ca9e9191", "LogicalResourceId": "stack1CustomResource4003511E", "NoEcho": false, "Data": {}} |
| 1698138073661 | [INFO] 2023-10-24T09:01:13.661Z b698df79-1457-4fdf-9c1e-5555b327c105 | status code: OK |
| 1698138073699 | END RequestId: b698df79-1457-4fdf-9c1e-5555b327c105 |
| 1698138073699 | REPORT RequestId: b698df79-1457-4fdf-9c1e-5555b327c105 Duration: 18572.32 ms Billed Duration: 18573 ms Memory Size: 128 MB Max Memory Used: 128 MB |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```
### Reproduction Steps
Create a new CDK app, and copy the following stack code:
```
import * as cdk from 'aws-cdk-lib';
import { BlockPublicAccess, Bucket } from 'aws-cdk-lib/aws-s3';
import { BucketDeployment, Source } from 'aws-cdk-lib/aws-s3-deployment';
import { Construct } from 'constructs';
import * as path from 'path';
export class S3DeployStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const bucket = new Bucket(this, "bucket", {
autoDeleteObjects: true,
removalPolicy: cdk.RemovalPolicy.DESTROY,
blockPublicAccess: BlockPublicAccess.BLOCK_ALL,
});
const s3deploy = new BucketDeployment(this, 'stack-1', {
destinationBucket: bucket,
sources: [Source.asset(path.join(__dirname, "../bin"))],
retainOnDelete: false,
})
}
}
```
Compile and deploy.
Next, change the `Bucket` resource `id` and redeploy.
The error is thrown and the stack goes into a failed rollback state from which one can only recover by destroying the stack.
### Possible Solution
### Additional Information/Context
_No response_
### CDK CLI Version
2.99.1 (build b2a895e)
### Framework Version
2.99.1
### Node.js Version
v20.2.0
### OS
MacOS
### Language
TypeScript
### Language Version
5.1.3
### Other information
_No response_
Contributor guide
Research direction
Start with the BucketDeployment custom resource Lambda and the reported update logs, focusing on the destination bucket change when retainOnDelete is false and autoDeleteObjects is true. Reproduce the deployment update from the issue and verify that changing the bucket id completes without a custom-resource failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100