AwsCustomResource: Race condition in IAM policy updates
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### What is the problem?
I have a construct that performs several AWS SDK calls using `AwsCustomResource`, each of which should have an IAM policy scoped as tightly as possible. Sometimes, later calls fail with a permission error, even though the IAM policy being applied is correct.
Per the documentation: "As this custom resource uses a singleton Lambda function, it's important to note the that function's role will eventually accumulate the permissions/grants from all resources."
What I discovered is that earlier custom resource calls succeed because it takes a little while (~60 seconds) for the Lambda to be created, allowing enough time for the associated IAM policy to propagate. However, subsequent custom resource calls reuse the Lambda, and if it executes too quickly after the policy is edited with the new permission, it won't have said permission and will fail.
### Reproduction Steps
I can provide code on request but can't yet post publicly (working on getting permission to do so). But to reproduce should be straightforward:
1. Create an AwsCustomResource call that relies on policy A. Execute that call.
2. Create an AwsCustomResource call that relies on policy B. Execute that call.
If the second call happens fast enough after the policy is applied, it will fail.
### What did you expect to happen?
All custom resource calls succeed.
### What actually happened?
One of the custom resource calls failed with a permissions error.
### CDK CLI Version
2.3.0 (build beaa5b2)
### Framework Version
_No response_
### Node.js Version
v16.7.0
### OS
MacOS
### Language
Typescript
### Language Version
4.5.4
### Other information
I have a CloudFormation event log I can share that illustrates the problem. Contact me directly and I can provide it.
Contributor guide
Research direction
Start with the AwsCustomResource singleton Lambda and the IAM policy update path described in the reproduction steps. Reproduce two calls using separate policies, then verify that the later call succeeds reliably after the policy update without a permissions error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- authorization, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100