aws / aws/aws-cdk

custom_resources: AwsCustomResource deadlocks sporadically

Open
#29,026 5 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/custom-resources bug effort/medium p3
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

We are using AwsCustomResource to execute s3:deleteBucketInventoryConfiguration on delete. We have observed sporadic failures that cause CloudFormation to end up hitting its one hour timeout because the custom resource never responds.

Looking in the logs for the custom resource lambda that CDK autogenerates, we see the task repeatedly time out after 120 seconds. There are no other logs. As per CloudTrail, the call to S3 is never even made.

### Expected Behavior

The custom resource should work.

### Current Behavior

It sporadically gets stuck and never responds.

### Reproduction Steps

```python
custom_resources.AwsCustomResource(
scope,
id,
install_latest_aws_sdk=True,
policy=custom_resources.AwsCustomResourcePolicy.from_statements([
iam.PolicyStatement(
actions=["s3:PutInventoryConfiguration"],
resources=[],
),
]),
on_delete={
"service": "S3",
"action": "deleteBucketInventoryConfiguration",
"parameters": {
"Bucket": ,
"Id": ,
}
}
)
```

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### CDK CLI Version

2.118.0 (build a40f2ec)

### Framework Version

_No response_

### Node.js Version

v20.10.0

### OS

Alpine 3.18

### Language

Python

### Language Version

3.19.0

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at the AwsCustomResource entry point and reproduce the on_delete call for S3 deleteBucketInventoryConfiguration with install_latest_aws_sdk enabled. Inspect the generated custom resource Lambda around the repeated 120-second task timeouts and verify that the resource responds reliably and the S3 call is reached.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.