aws-cloudformation / aws-cloudformation/cloudformation-coverage-roadmap

(custom-resources): onDelete is called before onCreate is done

Open
#1,730 0 comments 4 reactions 0 assignees View on GitHub
bug
Dominant language
No language data
Stars
1.1k
Forks
62
PR merge metrics
No merged PRs in 30d

Description

### Name of the resource

AWS::CloudFormation::CustomResource

### Resource Name

_No response_

### Issue Description

The custom resource lambda gets called with a delete event before the create event lambda ended.

When a stack containing a custom resource failed to deploy for reasons that are not related to the custom resource.
cloud formation canceled the creation of the custom resource ("Resource creation cancelled") and then called the lambda with a delete event.
This called can happen while a different instance of the lambda performs the create flow which can cause an unstable state.

### Expected Behavior

Cloud formation should wait until the create flow is done before starting the delete flow

### Observed Behavior

### CDK logs:

[**2023-06-26T19:14:29.759Z**] | 7:13:34 PM | **CREATE_IN_PROGRESS** | AWS::CloudFormation::CustomResource | (fixrds358createrdstableintegrationsCreateTableAndUser76462F51)

[**2023-06-26T19:14:29.762Z**] | 7:13:38 PM | **CREATE_FAILED** | AWS::CloudFormation::CustomResource | (fixrds358createrdstableintegrationsCreateTableAndUser76462F51) Resource creation cancelled

[**2023-06-26T19:14:29.762Z**] | 7:13:40 PM | **DELETE_IN_PROGRESS** | AWS::CloudFormation::CustomResource | (fixrds358createrdstableintegrationsCreateTableAndUser76462F51)

[**2023-06-26T19:14:29.769Z**] | 7:14:08 PM | **DELETE_COMPLETE** | AWS::CloudFormation::CustomResource | (fixrds358createrdstableintegrationsCreateTableAndUser76462F51)

### Cloud watch logs:
The create lambda invocation:
**2023-06-26T19:13:38.438Z** 7f69edde-d3f6-4dd1-ae40-086b4d765112 INFO { RequestType: 'Create', ...}
...
**2023-06-26T12:14:07.998-07:00** | REPORT RequestId: 7f69edde-d3f6-4dd1-ae40-086b4d765112

The delete lambda invocation:
**2023-06-26T12:13:44.680-07:00 | 2023-06-26T19:13:44.680Z** 406167ae-a7c6-4ce1-80bd-6d9b9dc2f2ba INFO { RequestType: 'Delete', ...}
...
**23-06-26T12:14:06.259-07:00** | REPORT RequestId: 406167ae-a7c6-4ce1-80bd-6d9b9dc2f2ba

Notice that the delete lambda start before the create lambda starts.

### Test Cases

Create a stack with two custom resource one that wait for a few seconds and failed and the other that wait much more.
The Second resource is expected to be called with onDelete before the onCreate is finished.

### Other Details

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the described stack with two AWS::CloudFormation::CustomResource resources, one failing after a short delay and the other running longer. Compare the CloudFormation and CloudWatch timelines for the Create and Delete events. Done means the lifecycle behavior is confirmed and the expected ordering or service-level limitation is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.