aws / aws/aws-cdk

cloudwatch: subscription filter creation is concurrent to CanInvokeLambda permission creation

Open
#23,177 16 comments 3 reactions 0 assignees View on GitHub
@aws-cdk/aws-cloudwatch bug effort/small p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

When we add subscription filter to the cloudwatch group log there are two operations: add permission to invoke lambda from cloudwatch service (CanInvokeLambda) and add subscription filter itself. The order is correct - add permission starts first and then create subscription filter. The problem is that adding permission can still be in progress and filter creation fails with the error "Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function"

### Expected Behavior

I expect deploy to wait permission creation, so filter creation will never fail.

### Current Behavior

Sometimes deploy succeeds, sometimes it fails

### Reproduction Steps

`````fsharp
logGroup.AddSubscriptionFilter("myid", SubscriptionFilterOptions
(
FilterPattern = FilterPattern.Any(
FilterPattern.StringValue("$.log_severity", "=", "ERROR"),
FilterPattern.StringValue("$.log_severity", "=", "WARN"),
FilterPattern.StringValue("$.log_severity", "=", "FATAL")
),
Destination = LambdaDestination(notificationLambda)
))
`````

### Possible Solution

Wait for permission creation

### Additional Information/Context

Amazon.CDK.Lib 2.44.0

### CDK CLI Version

2.50.0 (build 4c11af6)

### Framework Version

_No response_

### Node.js Version

18.12.1

### OS

Linux

### Language

.NET

### Language Version

_No response_

### Other information

failed log example:

```
30-Nov-2022 13:01:10 ec-m-tech-dev-common-MainStack | 16/80 | 1:01:07 PM | CREATE_IN_PROGRESS | AWS::Lambda::Permission | MainStack/DWEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId/CanInvokeLambda (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterIdCanInvokeLambda24FE7AF1)
30-Nov-2022 13:01:10 ec-m-tech-dev-common-MainStack | 16/80 | 1:01:08 PM | CREATE_IN_PROGRESS | AWS::Lambda::Permission | MainStack/DWEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId/CanInvokeLambda (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterIdCanInvokeLambda24FE7AF1) Resource creation Initiated
30-Nov-2022 13:01:10 ec-m-tech-dev-common-MainStack | 16/80 | 1:01:08 PM | CREATE_IN_PROGRESS | AWS::Logs::SubscriptionFilter | MainStack/DriveWealthEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterId0D1F460D) Resource creation Initiated
30-Nov-2022 13:01:10 ec-m-tech-dev-common-MainStack | 16/80 | 1:01:09 PM | CREATE_FAILED | AWS::Logs::SubscriptionFilter | MainStack/DWEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterId0D1F460D) Resource handler returned message: "Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function. (Service: CloudWatchLogs, Status Code: 400, Request ID: 3520c986-93b0-4c3e-a6d4-266bfad60a37)" (RequestToken: a31769e1-5aec-d85c-30ba-4c81e0eca8c1, HandlerErrorCode: InternalFailure)
30-Nov-2022 13:01:10 ec-m-tech-dev-common-MainStack | 16/80 | 1:01:09 PM | CREATE_FAILED | AWS::Lambda::Permission | MainStack/DWEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId/CanInvokeLambda (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterIdCanInvokeLambda24FE7AF1) Resource creation cancelled
```

Contributor guide

Open the contributing guide

Research direction

Start from the logGroup.AddSubscriptionFilter reproduction and inspect how the CloudWatch Logs subscription filter and Lambda CanInvokeLambda permission are represented during deployment. Reproduce the intermittent failure, then verify that deployment waits for permission creation before creating the filter and that the provided failure no longer occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.