aws / aws/aws-cdk

AwsCustomResource: Race condition in IAM policy updates

Open
#18,237 8 comments 7 reactions 0 assignees View on GitHub
@aws-cdk/aws-iam bug effort/medium p1
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### What is the problem?

I have a construct that performs several AWS SDK calls using `AwsCustomResource`, each of which should have an IAM policy scoped as tightly as possible. Sometimes, later calls fail with a permission error, even though the IAM policy being applied is correct.

Per the documentation: "As this custom resource uses a singleton Lambda function, it's important to note the that function's role will eventually accumulate the permissions/grants from all resources."

What I discovered is that earlier custom resource calls succeed because it takes a little while (~60 seconds) for the Lambda to be created, allowing enough time for the associated IAM policy to propagate. However, subsequent custom resource calls reuse the Lambda, and if it executes too quickly after the policy is edited with the new permission, it won't have said permission and will fail.

### Reproduction Steps

I can provide code on request but can't yet post publicly (working on getting permission to do so). But to reproduce should be straightforward:

1. Create an AwsCustomResource call that relies on policy A. Execute that call.
2. Create an AwsCustomResource call that relies on policy B. Execute that call.

If the second call happens fast enough after the policy is applied, it will fail.

### What did you expect to happen?

All custom resource calls succeed.

### What actually happened?

One of the custom resource calls failed with a permissions error.

### CDK CLI Version

2.3.0 (build beaa5b2)

### Framework Version

_No response_

### Node.js Version

v16.7.0

### OS

MacOS

### Language

Typescript

### Language Version

4.5.4

### Other information

I have a CloudFormation event log I can share that illustrates the problem. Contact me directly and I can provide it.

Contributor guide

Open the contributing guide

Research direction

Start with the AwsCustomResource singleton Lambda and the IAM policy update path described in the reproduction steps. Reproduce two calls using separate policies, then verify that the later call succeeds reliably after the policy update without a permissions error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
authorization, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.