aws / aws/aws-cdk

aws-logs: race condition with AWS Lambda default log groups

Open
#32,689 6 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-logs bug effort/small p3
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

#### my-stack.ts

```ts
const lambdaConfigs = [
{ lambda: backendLambda, name: "Backend" },
{ lambda: frontendLambda, name: "Frontend" },
{ lambda: clientNodeLambda, name: "ClientNode" },
{ lambda: clientPythonLambda, name: "ClientPython" },
{ lambda: clientRustLambda, name: "ClientRust" },
];

lambdaConfigs.forEach(({ lambda, name }) => {
const logGroup = new LogGroup(this, `${name}LambdaLogGroup`, {
logGroupName: `/aws/lambda/${lambda.functionName}`,
retention: RetentionDays.ONE_WEEK,
removalPolicy: RemovalPolicy.DESTROY,
});
});
```

In this scenario, all client functions are triggered by the EventBridge Scheduler every minute. During stack destruction via `cdk destroy`, there is a possibility that one of the Lambda functions could be invoked simultaneously. This results in the recreation of the default log group at `/aws/lambda/{function-name}`, even if it had just been deleted a few seconds earlier.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Last Known Working CDK Version

_No response_

### Expected Behavior

CDK should guarantee that AWS Lambda log groups are always deleted last to avoid race conditions.

### Current Behavior

![image](https://github.com/user-attachments/assets/3f703854-1235-4580-adf8-10f1fd38379a)

![image](https://github.com/user-attachments/assets/dd8fa7b4-41a2-4ebe-af5f-73e3712ba2e0)

One of the Log Groups seemed to survive the stack destruction despite having `RemovalPolicy.DESTROY` set. However, upon closer inspection, you can notice the retention period is set to `Never expire`, whereas in our stack we defined `RetentionDays.ONE_WEEK`. And that's because while the CDK was destroying the log group defined in our stack, the Lambda function got invoked, thus creating a new log group not managed by the CDK and with the default retention policy, which is `Never expire`.

### Reproduction Steps

see above.

### Possible Solution

One possible workaround is to create the log groups first, and assign them to each Lambda function directly at creation time.

```ts
const logGroupConfigs = [
{ name: "Backend", functionName: "backend-lambda" },
{ name: "Frontend", functionName: "frontend-lambda" },
{ name: "ClientNode", functionName: "client-node-lambda" },
{ name: "ClientPython", functionName: "client-python-lambda" },
{ name: "ClientRust", functionName: "client-rust-lambda" },
];

const logGroups = Object.fromEntries(
logGroupConfigs.map(({ name, functionName }) => [
name,
new LogGroup(this, `${name}LambdaLogGroup`, {
logGroupName: `/aws/lambda/${functionName}`,
retention: RetentionDays.ONE_WEEK,
removalPolicy: RemovalPolicy.DESTROY,
}),
]),
);

const clientNodeLambda = new NodejsFunction(this, "ClientNodeLambda", {
entry: join(__dirname, "..", "functions/client/node", "index.ts"),
runtime: Runtime.NODEJS_22_X,
architecture: Architecture.ARM_64,
memorySize: 1024,
timeout: Duration.minutes(1),
loggingFormat: LoggingFormat.JSON,
logGroup: logGroups.ClientNode,
});
```

However, this approach introduces additional complexity and makes the code less clean.

A more convenient solution would be to introduce a `Function.addLogGroup()` method, enabling users to link a log group to a Lambda function as a dependency within the same AWS CDK stack but in a different section of the code.

#### feature.ts

```ts
// Define the lambdas and their configurations
const lambdaConfigs = [
{ lambda: backendLambda, name: "Backend" },
{ lambda: frontendLambda, name: "Frontend" },
{ lambda: clientNodeLambda, name: "ClientNode" },
{ lambda: clientPythonLambda, name: "ClientPython" },
{ lambda: clientRustLambda, name: "ClientRust" },
];

// Create forwarder subscription for each lambda
lambdaConfigs.forEach(({ lambda, name }) => {
const logGroup = lambda.addLogGroup(
new LogGroup(this, `${name}LambdaLogGroup`, {
logGroupName: `/aws/lambda/${lambda.functionName}`,
retention: RetentionDays.ONE_WEEK,
removalPolicy: RemovalPolicy.DESTROY,
}),
);

new SubscriptionFilter(this, `${name}LambdaSubscription`, {
logGroup,
destination: new LambdaDestination(forwarderLambda),
filterPattern: FilterPattern.literal("{ $.__otel_otlp_stdout = * }"),
});
});
```

### Additional Information/Context

_No response_

### CDK CLI Version

2.173.4

### Framework Version

_No response_

### Node.js Version

22.12.0

### OS

Ubuntu 24.04.1

### Language

TypeScript

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the my-stack.ts reproduction and the feature.ts workaround, then inspect the Lambda and LogGroup resource lifecycle and dependency behavior in the CDK codebase. Confirm how stack destruction behaves when a Lambda is invoked concurrently, and define done as preventing an unmanaged default log group from being recreated while preserving the requested retention and removal policies.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.