aws-cdk-lib/aws-eks: waiter-state-machine is not authorized to access the Log Destination
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
Hello !
I am trying to create a cluster in an AWS Account I have admin permissions on.
The creation fails when a Provider I have no control on fails:
```logs
Failed resources:
Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26 | 4:05:50 PM | CREATE_FAILED | AWS::StepFunctions::StateMachine | @aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine (Providerwaiterstatemachine5D4A9DF0) Resource handler returned message: "The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID: c86629f3-fe7d-4c89-b782-f73b6bc410ed; Proxy: null)" (RequestToken: b9a6f357-3e31-2d56-ebc3-7f1d150459d3, HandlerErrorCode: AccessDenied)
Kubernetes | 4:06:08 PM | CREATE_FAILED | AWS::CloudFormation::Stack | @aws-cdk--aws-eks.ClusterResourceProvider.NestedStack/@aws-cdk--aws-eks.ClusterResourceProvider.NestedStackResource (awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourceProviderNestedStackResource9827C454) Embedded stack arn:aws:cloudformation:ap-northeast-1:797698216904:stack/Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26/b0fc7030-e521-11ef-a2f8-0ef66568cceb was not successfully created: The following resource(s) failed to create: [Providerwaiterstatemachine5D4A9DF0].
❌ Kubernetes failed: _ToolkitError: The stack named Kubernetes failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID: c86629f3-fe7d-4c89-b782-f73b6bc410ed; Proxy: null)" (RequestToken: b9a6f357-3e31-2d56-ebc3-7f1d150459d3, HandlerErrorCode: AccessDenied), Embedded stack arn:aws:cloudformation:ap-northeast-1:797698216904:stack/Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26/b0fc7030-e521-11ef-a2f8-0ef66568cceb was not successfully created: The following resource(s) failed to create: [Providerwaiterstatemachine5D4A9DF0].
```
How can I fix this as it seems to be out of my scope ?
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Last Known Working CDK Version
_No response_
### Expected Behavior
EKS cluster is created without error.
### Current Behavior
The creation fails when a Provider I have no control on fails:
```logs
Failed resources:
Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26 | 4:05:50 PM | CREATE_FAILED | AWS::StepFunctions::StateMachine | @aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine (Providerwaiterstatemachine5D4A9DF0) Resource handler returned message: "The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID: c86629f3-fe7d-4c89-b782-f73b6bc410ed; Proxy: null)" (RequestToken: b9a6f357-3e31-2d56-ebc3-7f1d150459d3, HandlerErrorCode: AccessDenied)
Kubernetes | 4:06:08 PM | CREATE_FAILED | AWS::CloudFormation::Stack | @aws-cdk--aws-eks.ClusterResourceProvider.NestedStack/@aws-cdk--aws-eks.ClusterResourceProvider.NestedStackResource (awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourceProviderNestedStackResource9827C454) Embedded stack arn:aws:cloudformation:ap-northeast-1:797698216904:stack/Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26/b0fc7030-e521-11ef-a2f8-0ef66568cceb was not successfully created: The following resource(s) failed to create: [Providerwaiterstatemachine5D4A9DF0].
❌ Kubernetes failed: _ToolkitError: The stack named Kubernetes failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID: c86629f3-fe7d-4c89-b782-f73b6bc410ed; Proxy: null)" (RequestToken: b9a6f357-3e31-2d56-ebc3-7f1d150459d3, HandlerErrorCode: AccessDenied), Embedded stack arn:aws:cloudformation:ap-northeast-1:797698216904:stack/Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26/b0fc7030-e521-11ef-a2f8-0ef66568cceb was not successfully created: The following resource(s) failed to create: [Providerwaiterstatemachine5D4A9DF0].
```
### Reproduction Steps
I have the following package.json:
```json
{
"name": "cdk",
"bin": {
"cdk": "bin/cdk.js"
},
"scripts": {
"build": "tsc",
"watch": "tsc -w",
"test": "jest",
"cdk": "cdk"
},
"devDependencies": {
"@types/jest": "^29.5.14",
"@types/semver": "^7.5.8",
"aws-cdk": "2.178.1",
"jest": "^29.7.0",
"ts-jest": "^29.2.5",
"ts-node": "^10.9.2",
"typescript": "~5.6.3"
},
"dependencies": {
"@aws-cdk/lambda-layer-kubectl-v32": "2.0.0",
"aws-cdk-lib": "2.178.1",
"constructs": "^10.0.0",
"semver": "^7.7.1"
}
}
```
And here is a kind of simplified version of my Stack
```typescript
import { KubectlV32Layer } from "@aws-cdk/lambda-layer-kubectl-v32";
import { Stack, StackProps } from "aws-cdk-lib";
import { IVpc, SubnetType } from "aws-cdk-lib/aws-ec2";
import {
Cluster,
ClusterLoggingTypes,
EndpointAccess,
KubectlProvider,
KubernetesVersion,
} from "aws-cdk-lib/aws-eks";
import { Construct } from "constructs";
import {
AccountRootPrincipal,
Effect,
Group,
IRole,
ManagedPolicy,
PolicyStatement,
Role,
} from "aws-cdk-lib/aws-iam";
const kubernetesVersion = "1.32";
export class EksCluster extends Stack {
public readonly cluster: Cluster;
public readonly kubernetesVersion: KubernetesVersion;
constructor(
scope: Construct,
id: string,
props: StackProps & {
vpc: IVpc;
}
) {
super(scope, id, props);
const { vpc, awsAccountProduction } = props;
this.kubernetesVersion = KubernetesVersion.of(kubernetesVersion);
const adminRole = this.makeIamRole("Admin");
this.cluster = this.makeCluster({ vpc, mastersRole: adminRole });
}
private makeIamRole(level: "Admin") {
const role = new Role(this, `${level}Role`, {
assumedBy: new AccountRootPrincipal(),
roleName: `eks-${level.toLowerCase()}`,
});
const policyStatement = new PolicyStatement({
resources: [role.roleArn],
actions: ["sts:AssumeRole"],
effect: Effect.ALLOW,
});
const managedPolicy = new ManagedPolicy(this, `${level}Policy`, {
managedPolicyName: `assume-eks-${level.toLocaleLowerCase()}-role`,
});
managedPolicy.addStatements(policyStatement);
return role;
}
private makeCluster({
vpc,
mastersRole,
}: {
vpc: IVpc;
mastersRole?: IRole;
}): Cluster {
const cluster = new Cluster(this, "Cluster", {
clusterName: "eks-default",
vpc,
vpcSubnets: [{ subnetType: SubnetType.PRIVATE_WITH_EGRESS }],
defaultCapacity: 0,
version: this.kubernetesVersion,
kubectlLayer: new KubectlV32Layer(this, "kubectl"),
mastersRole,
clusterLogging: [
ClusterLoggingTypes.API,
ClusterLoggingTypes.AUTHENTICATOR,
ClusterLoggingTypes.CONTROLLER_MANAGER,
ClusterLoggingTypes.SCHEDULER,
],
});
return cluster;
}
```
Portion of the generated template:
```json
"ProviderwaiterstatemachineRoleDefaultPolicyD3C3DA1A": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyDocument": {
"Statement": [
{
"Action": "lambda:InvokeFunction",
"Effect": "Allow",
"Resource": [
{
"Fn::GetAtt": [
"ProviderframeworkisComplete26D7B0CB",
"Arn"
]
},
{
"Fn::GetAtt": [
"ProviderframeworkonTimeout0B47CA38",
"Arn"
]
},
{
"Fn::Join": [
"",
[
{
"Fn::GetAtt": [
"ProviderframeworkisComplete26D7B0CB",
"Arn"
]
},
":*"
]
]
},
{
"Fn::Join": [
"",
[
{
"Fn::GetAtt": [
"ProviderframeworkonTimeout0B47CA38",
"Arn"
]
},
":*"
]
]
}
]
},
{
"Action": [
"logs:CreateLogDelivery",
"logs:CreateLogStream",
"logs:DeleteLogDelivery",
"logs:DescribeLogGroups",
"logs:DescribeResourcePolicies",
"logs:GetLogDelivery",
"logs:ListLogDeliveries",
"logs:PutLogEvents",
"logs:PutResourcePolicy",
"logs:UpdateLogDelivery"
],
"Effect": "Allow",
"Resource": "*"
}
],
"Version": "2012-10-17"
},
"PolicyName": "ProviderwaiterstatemachineRoleDefaultPolicyD3C3DA1A",
"Roles": [
{
"Ref": "ProviderwaiterstatemachineRole0C7159F9"
}
]
},
"Metadata": {
"aws:cdk:path": "Kubernetes/@aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine/Role/DefaultPolicy/Resource"
}
},
"ProviderwaiterstatemachineLogGroupDD693A98": {
"Type": "AWS::Logs::LogGroup",
"Properties": {
"LogGroupName": {
"Fn::Join": [
"",
[
"/aws/vendedlogs/states/waiter-state-machine-",
{
"Ref": "ProviderframeworkisComplete26D7B0CB"
},
"-c8100e0c94c82f842732ba88b108b9eee575b910e3"
]
]
},
"RetentionInDays": 731,
"Tags": [
{
"Key": "CmBillingGroup",
"Value": "CRM Shared"
}
]
},
"UpdateReplacePolicy": "Retain",
"DeletionPolicy": "Retain",
"Metadata": {
"aws:cdk:path": "Kubernetes/@aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine/LogGroup/Resource"
}
},
```
### Possible Solution
_No response_
### Additional Information/Context
I saw people say that the resource policies for logs have a limited size but I am not sure if that is the issue as the logs are in `/aws/vendedlogs/states/`:
```bash
aws logs describe-resource-policies | wc
54 85 4575
```
### CDK CLI Version
2.178.1 (build ae342cb)
### Framework Version
_No response_
### Node.js Version
v22.11.0
### OS
macOS Version 15.2 (24C101)
### Language
TypeScript
### Language Version
5.6.3
### Other information
_No response_
Contributor guide
Research direction
Start by reproducing the failure with aws-cdk-lib 2.178.1 and inspect the generated template for @aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine, especially its IAM policy and LogGroup. Trace how the waiter state machine configures its log destination and verify the generated permissions against the AccessDenied error; done means the EKS cluster provider creates successfully without the Step Functions log authorization failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, typescript
- Domain
- cloud, infrastructure, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100