aws / aws/aws-cdk

(custom-resources): onDelete is called before onCreate is done

Open
#26,139 1 comment 1 reaction 0 assignees View on GitHub
@aws-cdk/custom-resources blocked bug needs-cfn p1
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
1d 19h
Merged PRs (30d)
71

Description

### Describe the bug

The custom resource lambda gets called with a delete event before the create event lambda ended.

When a stack containing a custom resource failed to deploy for reasons that are not related to the custom resource.
cloud formation canceled the creation of the custom resource ("Resource creation cancelled") and then called the lambda with a delete event.
This called can happen while a different instance of the lambda performs the create flow which can cause an unstable state.

### Expected Behavior

CDK will wait until the create flow is done before starting the delete flow

### Current Behavior

### CDK logs:

[**2023-06-26T19:14:29.759Z**] | 7:13:34 PM | **CREATE_IN_PROGRESS** | AWS::CloudFormation::CustomResource | (fixrds358createrdstableintegrationsCreateTableAndUser76462F51)

[**2023-06-26T19:14:29.762Z**] | 7:13:38 PM | **CREATE_FAILED** | AWS::CloudFormation::CustomResource | (fixrds358createrdstableintegrationsCreateTableAndUser76462F51) Resource creation cancelled

[**2023-06-26T19:14:29.762Z**] | 7:13:40 PM | **DELETE_IN_PROGRESS** | AWS::CloudFormation::CustomResource | (fixrds358createrdstableintegrationsCreateTableAndUser76462F51)

[**2023-06-26T19:14:29.769Z**] | 7:14:08 PM | **DELETE_COMPLETE** | AWS::CloudFormation::CustomResource | (fixrds358createrdstableintegrationsCreateTableAndUser76462F51)

### cloud watch logs:
The create lambda invocation:
**2023-06-26T19:13:38.438Z** 7f69edde-d3f6-4dd1-ae40-086b4d765112 INFO { RequestType: 'Create', ...}
...
**2023-06-26T12:14:07.998-07:00** | REPORT RequestId: 7f69edde-d3f6-4dd1-ae40-086b4d765112

The delete lambda invocation:
**2023-06-26T12:13:44.680-07:00 | 2023-06-26T19:13:44.680Z** 406167ae-a7c6-4ce1-80bd-6d9b9dc2f2ba INFO { RequestType: 'Delete', ...}
...
**23-06-26T12:14:06.259-07:00** | REPORT RequestId: 406167ae-a7c6-4ce1-80bd-6d9b9dc2f2ba

Notice that the delete lambda start before the create lambda starts.

### Reproduction Steps

I will try to create one soon

### Possible Solution

Do not initiate the custom resource delete flow until it was created.

### Additional Information/Context

_No response_

### CDK CLI Version

2.75.0

### Framework Version

_No response_

### Node.js Version

18

### OS

Ubuntu

### Language

Typescript

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing the AWS CDK custom resource Lambda invocation flow for Create and Delete events, using the provided CloudFormation and CloudWatch timelines as the failing scenario. Look for existing custom-resource tests or the relevant handler entry point, then establish a regression test showing that Delete does not begin before Create finishes. Done means the reported cancellation sequence no longer overlaps the two flows.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.