aws / aws/aws-cdk

s3: possible race condition, adding new object to bucket between deletion of bucket objects and bucket deletion prevents deletion

Open
#21,776 20 comments 2 reactions 1 assignee Claimed by @GavinZZ View on GitHub
@aws-cdk/aws-s3 @aws-cdk/custom-resources bug documentation p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

If I create a log bucket. Then a content bucket, using log bucket for server access logs, when I do a "cdk destroy" the access logs are not removed and the stack fails to destroy.

### Expected Behavior

The destroy to complete successfully! (The log bucket and content to be removed.)

### Current Behavior

Cloudwatch logs for the delete objects Lambda suggests it was successful :
```
START RequestId: 72426a95-6f49-4d26-9642-2e15450b205a Version: $LATEST
2022-08-26T10:14:21.449Z 72426a95-6f49-4d26-9642-2e15450b205a INFO
{
"RequestType": "Delete",
"ServiceToken": "arn:aws:lambda:eu-west-2:123492961234:function:CdkBucketStack-CustomS3AutoDeleteObjectsCustomReso-kArk0g4I9Il2",
"ResponseURL": "...",
"StackId": "arn:aws:cloudformation:eu-west-2:123492961234:stack/CdkBucketStack/0531d0e0-2522-11ed-abbb-0aa24eb467d4",
"RequestId": "b81d7b6f-c28c-455f-83cf-8cf4be357191",
"LogicalResourceId": "LogBucketAutoDeleteObjectsCustomResource7762F42C",
"PhysicalResourceId": "f9dbbbc0-641f-4026-a15e-df345405a307",
"ResourceType": "Custom::S3AutoDeleteObjects",
"ResourceProperties": {
"ServiceToken": "arn:aws:lambda:eu-west-2:123492961234:function:CdkBucketStack-CustomS3AutoDeleteObjectsCustomReso-kArk0g4I9Il2",
"BucketName": "cdkbucketstack-logbucketcc3b17e8-79kbay52r42q"
}
}

2022-08-26T10:14:27.491Z 72426a95-6f49-4d26-9642-2e15450b205a INFO submit response to cloudformation {
Status: 'SUCCESS',
Reason: 'SUCCESS',
StackId: 'arn:aws:cloudformation:eu-west-2:123492961234:stack/CdkBucketStack/0531d0e0-2522-11ed-abbb-0aa24eb467d4',
RequestId: 'b81d7b6f-c28c-455f-83cf-8cf4be357191',
PhysicalResourceId: 'f9dbbbc0-641f-4026-a15e-df345405a307',
LogicalResourceId: 'LogBucketAutoDeleteObjectsCustomResource7762F42C',
NoEcho: undefined,
Data: undefined
}
END RequestId: 72426a95-6f49-4d26-9642-2e15450b205a
REPORT RequestId: 72426a95-6f49-4d26-9642-2e15450b205a Duration: 6312.29 ms Billed Duration: 6313 ms Memory Size: 128 MB Max Memory Used: 81 MB Init Duration: 174.79 ms
```

But the result of the `cdk destroy` is a fail because there is content :

```
CdkBucketStack: destroying...
11:15:21 | DELETE_FAILED | AWS::S3::Bucket | LogBucketCC3B17E8
The bucket you tried to delete is not empty (Service: Amazon S3; Status Code: 409; Error Code: BucketNotEmpty; Request ID: HJVZC3J1DP4N2QSM; S3 Extended Request ID: /OcCq7vdx6Yxt6bShdNQhgr8fUI3/pG1gmCuC4cu7dBm3juB+Sa9U+FM+O41vk+FVCV8Xwk49YA=; Proxy: null)

❌ CdkBucketStack: destroy failed Error: The stack named CdkBucketStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [LogBucketCC3B17E8]. ): The bucket you tried to delete is not empty (Service: Amazon S3; Status Code: 409; Error Code: BucketNotEmpty; Request ID: HJVZC3J1DP4N2QSM; S3 Extended Request ID: /OcCq7vdx6Yxt6bShdNQhgr8fUI3/pG1gmCuC4cu7dBm3juB+Sa9U+FM+O41vk+FVCV8Xwk49YA=; Proxy: null)
```

### Reproduction Steps

Deploy the template below. Upload your current directory to "bucket" and wait up to an hour until there are some logs in logBucket. Then run the destroy.

(Yes the code is a bit janky, I did a lazy copy/paste and removed some logic that isn't relevant etc...)

```
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as iam from 'aws-cdk-lib/aws-iam';

export class CdkBucketStack extends cdk.Stack {
public readonly Bucket: cdk.CfnOutput;
public readonly logBucket: cdk.CfnOutput;
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
// Create a bucket for all the different logs.
let logBucketProps: any = {
removalPolicy: cdk.RemovalPolicy.DESTROY,
autoDeleteObjects: true,
accessControl: s3.BucketAccessControl.LOG_DELIVERY_WRITE,
};
logBucketProps["enforceSSL"] = true;
logBucketProps["encryption"] = s3.BucketEncryption.S3_MANAGED;
logBucketProps["blockPublicAccess"] = s3.BlockPublicAccess.BLOCK_ALL;
const logBucket = new s3.Bucket(this, 'LogBucket', logBucketProps);
logBucket.addToResourcePolicy(
new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
principals: [new iam.ServicePrincipal('logging.s3.amazonaws.com')],
actions: ['s3:PutObject'],
resources: [`${logBucket.bucketArn}/*`],
}),
);

let BucketProps: any = {
removalPolicy: cdk.RemovalPolicy.DESTROY,
autoDeleteObjects: true,
};
BucketProps["enforceSSL"] = true;
BucketProps["encryption"] = s3.BucketEncryption.S3_MANAGED;
BucketProps["serverAccessLogsPrefix"] = "output-access-logs/";
BucketProps["serverAccessLogsBucket"] = logBucket;
BucketProps["blockPublicAccess"] = s3.BlockPublicAccess.BLOCK_ALL;
const bucket = new s3.Bucket(this, 'OutputBucket', BucketProps);

this.Bucket = new cdk.CfnOutput(this, 'S3Bucket', {
value: bucket.bucketName,
description: "The bucket",
});
this.logBucket = new cdk.CfnOutput(this, 'logS3Bucket', {
value: logBucket.bucketName,
description: "The log bucket",
});
}
}
```

### Possible Solution

I imagine the "delete content" lambda is not checking some error condition it isn't expecting and reporting success when it hasn't succeeded. The Lambda does get deleted and the code in the Lambda editor is minified so I couldn't easily retry or debug it.

### Additional Information/Context

The logic I removed in the code example avoids deleting the content in our "production" deployments. But in dev, we have no need to keep the access logs. So if someone was thinking "but it's log data, we can't delete that", please : do delete _any_ data. Or at least provide a "actuallyAutoDeleteAllObjects: true" option.

### CDK CLI Version

2.38.1 (build a5ced21)

### Framework Version

_No response_

### Node.js Version

Node.js v18.8.0

### OS

OSX

### Language

Typescript

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.