cloudwatch: subscription filter creation is concurrent to CanInvokeLambda permission creation
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
When we add subscription filter to the cloudwatch group log there are two operations: add permission to invoke lambda from cloudwatch service (CanInvokeLambda) and add subscription filter itself. The order is correct - add permission starts first and then create subscription filter. The problem is that adding permission can still be in progress and filter creation fails with the error "Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function"
### Expected Behavior
I expect deploy to wait permission creation, so filter creation will never fail.
### Current Behavior
Sometimes deploy succeeds, sometimes it fails
### Reproduction Steps
`````fsharp
logGroup.AddSubscriptionFilter("myid", SubscriptionFilterOptions
(
FilterPattern = FilterPattern.Any(
FilterPattern.StringValue("$.log_severity", "=", "ERROR"),
FilterPattern.StringValue("$.log_severity", "=", "WARN"),
FilterPattern.StringValue("$.log_severity", "=", "FATAL")
),
Destination = LambdaDestination(notificationLambda)
))
`````
### Possible Solution
Wait for permission creation
### Additional Information/Context
Amazon.CDK.Lib 2.44.0
### CDK CLI Version
2.50.0 (build 4c11af6)
### Framework Version
_No response_
### Node.js Version
18.12.1
### OS
Linux
### Language
.NET
### Language Version
_No response_
### Other information
failed log example:
```
30-Nov-2022 13:01:10 ec-m-tech-dev-common-MainStack | 16/80 | 1:01:07 PM | CREATE_IN_PROGRESS | AWS::Lambda::Permission | MainStack/DWEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId/CanInvokeLambda (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterIdCanInvokeLambda24FE7AF1)
30-Nov-2022 13:01:10 ec-m-tech-dev-common-MainStack | 16/80 | 1:01:08 PM | CREATE_IN_PROGRESS | AWS::Lambda::Permission | MainStack/DWEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId/CanInvokeLambda (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterIdCanInvokeLambda24FE7AF1) Resource creation Initiated
30-Nov-2022 13:01:10 ec-m-tech-dev-common-MainStack | 16/80 | 1:01:08 PM | CREATE_IN_PROGRESS | AWS::Logs::SubscriptionFilter | MainStack/DriveWealthEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterId0D1F460D) Resource creation Initiated
30-Nov-2022 13:01:10 ec-m-tech-dev-common-MainStack | 16/80 | 1:01:09 PM | CREATE_FAILED | AWS::Logs::SubscriptionFilter | MainStack/DWEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterId0D1F460D) Resource handler returned message: "Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function. (Service: CloudWatchLogs, Status Code: 400, Request ID: 3520c986-93b0-4c3e-a6d4-266bfad60a37)" (RequestToken: a31769e1-5aec-d85c-30ba-4c81e0eca8c1, HandlerErrorCode: InternalFailure)
30-Nov-2022 13:01:10 ec-m-tech-dev-common-MainStack | 16/80 | 1:01:09 PM | CREATE_FAILED | AWS::Lambda::Permission | MainStack/DWEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId/CanInvokeLambda (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterIdCanInvokeLambda24FE7AF1) Resource creation cancelled
```
Contributor guide
Research direction
Start from the logGroup.AddSubscriptionFilter reproduction and inspect how the CloudWatch Logs subscription filter and Lambda CanInvokeLambda permission are represented during deployment. Reproduce the intermittent failure, then verify that deployment waits for permission creation before creating the filter and that the provided failure no longer occurs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100