aws-logs: race condition with AWS Lambda default log groups
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
#### my-stack.ts
```ts
const lambdaConfigs = [
{ lambda: backendLambda, name: "Backend" },
{ lambda: frontendLambda, name: "Frontend" },
{ lambda: clientNodeLambda, name: "ClientNode" },
{ lambda: clientPythonLambda, name: "ClientPython" },
{ lambda: clientRustLambda, name: "ClientRust" },
];
lambdaConfigs.forEach(({ lambda, name }) => {
const logGroup = new LogGroup(this, `${name}LambdaLogGroup`, {
logGroupName: `/aws/lambda/${lambda.functionName}`,
retention: RetentionDays.ONE_WEEK,
removalPolicy: RemovalPolicy.DESTROY,
});
});
```
In this scenario, all client functions are triggered by the EventBridge Scheduler every minute. During stack destruction via `cdk destroy`, there is a possibility that one of the Lambda functions could be invoked simultaneously. This results in the recreation of the default log group at `/aws/lambda/{function-name}`, even if it had just been deleted a few seconds earlier.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Last Known Working CDK Version
_No response_
### Expected Behavior
CDK should guarantee that AWS Lambda log groups are always deleted last to avoid race conditions.
### Current Behavior


One of the Log Groups seemed to survive the stack destruction despite having `RemovalPolicy.DESTROY` set. However, upon closer inspection, you can notice the retention period is set to `Never expire`, whereas in our stack we defined `RetentionDays.ONE_WEEK`. And that's because while the CDK was destroying the log group defined in our stack, the Lambda function got invoked, thus creating a new log group not managed by the CDK and with the default retention policy, which is `Never expire`.
### Reproduction Steps
see above.
### Possible Solution
One possible workaround is to create the log groups first, and assign them to each Lambda function directly at creation time.
```ts
const logGroupConfigs = [
{ name: "Backend", functionName: "backend-lambda" },
{ name: "Frontend", functionName: "frontend-lambda" },
{ name: "ClientNode", functionName: "client-node-lambda" },
{ name: "ClientPython", functionName: "client-python-lambda" },
{ name: "ClientRust", functionName: "client-rust-lambda" },
];
const logGroups = Object.fromEntries(
logGroupConfigs.map(({ name, functionName }) => [
name,
new LogGroup(this, `${name}LambdaLogGroup`, {
logGroupName: `/aws/lambda/${functionName}`,
retention: RetentionDays.ONE_WEEK,
removalPolicy: RemovalPolicy.DESTROY,
}),
]),
);
const clientNodeLambda = new NodejsFunction(this, "ClientNodeLambda", {
entry: join(__dirname, "..", "functions/client/node", "index.ts"),
runtime: Runtime.NODEJS_22_X,
architecture: Architecture.ARM_64,
memorySize: 1024,
timeout: Duration.minutes(1),
loggingFormat: LoggingFormat.JSON,
logGroup: logGroups.ClientNode,
});
```
However, this approach introduces additional complexity and makes the code less clean.
A more convenient solution would be to introduce a `Function.addLogGroup()` method, enabling users to link a log group to a Lambda function as a dependency within the same AWS CDK stack but in a different section of the code.
#### feature.ts
```ts
// Define the lambdas and their configurations
const lambdaConfigs = [
{ lambda: backendLambda, name: "Backend" },
{ lambda: frontendLambda, name: "Frontend" },
{ lambda: clientNodeLambda, name: "ClientNode" },
{ lambda: clientPythonLambda, name: "ClientPython" },
{ lambda: clientRustLambda, name: "ClientRust" },
];
// Create forwarder subscription for each lambda
lambdaConfigs.forEach(({ lambda, name }) => {
const logGroup = lambda.addLogGroup(
new LogGroup(this, `${name}LambdaLogGroup`, {
logGroupName: `/aws/lambda/${lambda.functionName}`,
retention: RetentionDays.ONE_WEEK,
removalPolicy: RemovalPolicy.DESTROY,
}),
);
new SubscriptionFilter(this, `${name}LambdaSubscription`, {
logGroup,
destination: new LambdaDestination(forwarderLambda),
filterPattern: FilterPattern.literal("{ $.__otel_otlp_stdout = * }"),
});
});
```
### Additional Information/Context
_No response_
### CDK CLI Version
2.173.4
### Framework Version
_No response_
### Node.js Version
22.12.0
### OS
Ubuntu 24.04.1
### Language
TypeScript
### Language Version
_No response_
### Other information
_No response_
Contributor guide
Research direction
Start with the my-stack.ts reproduction and the feature.ts workaround, then inspect the Lambda and LogGroup resource lifecycle and dependency behavior in the CDK codebase. Confirm how stack destruction behaves when a Lambda is invoked concurrently, and define done as preventing an unmanaged default log group from being recreated while preserving the requested retention and removal policies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100