aws / aws/aws-cdk

aws-cdk-lib/aws-eks: waiter-state-machine is not authorized to access the Log Destination

Open
#33,332 3 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-iam bug effort/medium p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

Hello !

I am trying to create a cluster in an AWS Account I have admin permissions on.

The creation fails when a Provider I have no control on fails:
```logs
Failed resources:
Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26 | 4:05:50 PM | CREATE_FAILED | AWS::StepFunctions::StateMachine | @aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine (Providerwaiterstatemachine5D4A9DF0) Resource handler returned message: "The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID: c86629f3-fe7d-4c89-b782-f73b6bc410ed; Proxy: null)" (RequestToken: b9a6f357-3e31-2d56-ebc3-7f1d150459d3, HandlerErrorCode: AccessDenied)
Kubernetes | 4:06:08 PM | CREATE_FAILED | AWS::CloudFormation::Stack | @aws-cdk--aws-eks.ClusterResourceProvider.NestedStack/@aws-cdk--aws-eks.ClusterResourceProvider.NestedStackResource (awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourceProviderNestedStackResource9827C454) Embedded stack arn:aws:cloudformation:ap-northeast-1:797698216904:stack/Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26/b0fc7030-e521-11ef-a2f8-0ef66568cceb was not successfully created: The following resource(s) failed to create: [Providerwaiterstatemachine5D4A9DF0].
❌ Kubernetes failed: _ToolkitError: The stack named Kubernetes failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID: c86629f3-fe7d-4c89-b782-f73b6bc410ed; Proxy: null)" (RequestToken: b9a6f357-3e31-2d56-ebc3-7f1d150459d3, HandlerErrorCode: AccessDenied), Embedded stack arn:aws:cloudformation:ap-northeast-1:797698216904:stack/Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26/b0fc7030-e521-11ef-a2f8-0ef66568cceb was not successfully created: The following resource(s) failed to create: [Providerwaiterstatemachine5D4A9DF0].
```

How can I fix this as it seems to be out of my scope ?

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Last Known Working CDK Version

_No response_

### Expected Behavior

EKS cluster is created without error.

### Current Behavior

The creation fails when a Provider I have no control on fails:
```logs
Failed resources:
Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26 | 4:05:50 PM | CREATE_FAILED | AWS::StepFunctions::StateMachine | @aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine (Providerwaiterstatemachine5D4A9DF0) Resource handler returned message: "The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID: c86629f3-fe7d-4c89-b782-f73b6bc410ed; Proxy: null)" (RequestToken: b9a6f357-3e31-2d56-ebc3-7f1d150459d3, HandlerErrorCode: AccessDenied)
Kubernetes | 4:06:08 PM | CREATE_FAILED | AWS::CloudFormation::Stack | @aws-cdk--aws-eks.ClusterResourceProvider.NestedStack/@aws-cdk--aws-eks.ClusterResourceProvider.NestedStackResource (awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourceProviderNestedStackResource9827C454) Embedded stack arn:aws:cloudformation:ap-northeast-1:797698216904:stack/Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26/b0fc7030-e521-11ef-a2f8-0ef66568cceb was not successfully created: The following resource(s) failed to create: [Providerwaiterstatemachine5D4A9DF0].
❌ Kubernetes failed: _ToolkitError: The stack named Kubernetes failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID: c86629f3-fe7d-4c89-b782-f73b6bc410ed; Proxy: null)" (RequestToken: b9a6f357-3e31-2d56-ebc3-7f1d150459d3, HandlerErrorCode: AccessDenied), Embedded stack arn:aws:cloudformation:ap-northeast-1:797698216904:stack/Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26/b0fc7030-e521-11ef-a2f8-0ef66568cceb was not successfully created: The following resource(s) failed to create: [Providerwaiterstatemachine5D4A9DF0].
```

### Reproduction Steps

I have the following package.json:
```json
{
"name": "cdk",
"bin": {
"cdk": "bin/cdk.js"
},
"scripts": {
"build": "tsc",
"watch": "tsc -w",
"test": "jest",
"cdk": "cdk"
},
"devDependencies": {
"@types/jest": "^29.5.14",
"@types/semver": "^7.5.8",
"aws-cdk": "2.178.1",
"jest": "^29.7.0",
"ts-jest": "^29.2.5",
"ts-node": "^10.9.2",
"typescript": "~5.6.3"
},
"dependencies": {
"@aws-cdk/lambda-layer-kubectl-v32": "2.0.0",
"aws-cdk-lib": "2.178.1",
"constructs": "^10.0.0",
"semver": "^7.7.1"
}
}
```

And here is a kind of simplified version of my Stack
```typescript
import { KubectlV32Layer } from "@aws-cdk/lambda-layer-kubectl-v32";
import { Stack, StackProps } from "aws-cdk-lib";
import { IVpc, SubnetType } from "aws-cdk-lib/aws-ec2";
import {
Cluster,
ClusterLoggingTypes,
EndpointAccess,
KubectlProvider,
KubernetesVersion,
} from "aws-cdk-lib/aws-eks";
import { Construct } from "constructs";
import {
AccountRootPrincipal,
Effect,
Group,
IRole,
ManagedPolicy,
PolicyStatement,
Role,
} from "aws-cdk-lib/aws-iam";

const kubernetesVersion = "1.32";

export class EksCluster extends Stack {
public readonly cluster: Cluster;
public readonly kubernetesVersion: KubernetesVersion;

constructor(
scope: Construct,
id: string,
props: StackProps & {
vpc: IVpc;
}
) {
super(scope, id, props);
const { vpc, awsAccountProduction } = props;
this.kubernetesVersion = KubernetesVersion.of(kubernetesVersion);
const adminRole = this.makeIamRole("Admin");
this.cluster = this.makeCluster({ vpc, mastersRole: adminRole });
}

private makeIamRole(level: "Admin") {
const role = new Role(this, `${level}Role`, {
assumedBy: new AccountRootPrincipal(),
roleName: `eks-${level.toLowerCase()}`,
});
const policyStatement = new PolicyStatement({
resources: [role.roleArn],
actions: ["sts:AssumeRole"],
effect: Effect.ALLOW,
});
const managedPolicy = new ManagedPolicy(this, `${level}Policy`, {
managedPolicyName: `assume-eks-${level.toLocaleLowerCase()}-role`,
});
managedPolicy.addStatements(policyStatement);
return role;
}

private makeCluster({
vpc,
mastersRole,
}: {
vpc: IVpc;
mastersRole?: IRole;
}): Cluster {
const cluster = new Cluster(this, "Cluster", {
clusterName: "eks-default",
vpc,
vpcSubnets: [{ subnetType: SubnetType.PRIVATE_WITH_EGRESS }],
defaultCapacity: 0,
version: this.kubernetesVersion,
kubectlLayer: new KubectlV32Layer(this, "kubectl"),
mastersRole,
clusterLogging: [
ClusterLoggingTypes.API,
ClusterLoggingTypes.AUTHENTICATOR,
ClusterLoggingTypes.CONTROLLER_MANAGER,
ClusterLoggingTypes.SCHEDULER,
],
});
return cluster;
}
```

Portion of the generated template:
```json

"ProviderwaiterstatemachineRoleDefaultPolicyD3C3DA1A": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyDocument": {
"Statement": [
{
"Action": "lambda:InvokeFunction",
"Effect": "Allow",
"Resource": [
{
"Fn::GetAtt": [
"ProviderframeworkisComplete26D7B0CB",
"Arn"
]
},
{
"Fn::GetAtt": [
"ProviderframeworkonTimeout0B47CA38",
"Arn"
]
},
{
"Fn::Join": [
"",
[
{
"Fn::GetAtt": [
"ProviderframeworkisComplete26D7B0CB",
"Arn"
]
},
":*"
]
]
},
{
"Fn::Join": [
"",
[
{
"Fn::GetAtt": [
"ProviderframeworkonTimeout0B47CA38",
"Arn"
]
},
":*"
]
]
}
]
},
{
"Action": [
"logs:CreateLogDelivery",
"logs:CreateLogStream",
"logs:DeleteLogDelivery",
"logs:DescribeLogGroups",
"logs:DescribeResourcePolicies",
"logs:GetLogDelivery",
"logs:ListLogDeliveries",
"logs:PutLogEvents",
"logs:PutResourcePolicy",
"logs:UpdateLogDelivery"
],
"Effect": "Allow",
"Resource": "*"
}
],
"Version": "2012-10-17"
},
"PolicyName": "ProviderwaiterstatemachineRoleDefaultPolicyD3C3DA1A",
"Roles": [
{
"Ref": "ProviderwaiterstatemachineRole0C7159F9"
}
]
},
"Metadata": {
"aws:cdk:path": "Kubernetes/@aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine/Role/DefaultPolicy/Resource"
}
},
"ProviderwaiterstatemachineLogGroupDD693A98": {
"Type": "AWS::Logs::LogGroup",
"Properties": {
"LogGroupName": {
"Fn::Join": [
"",
[
"/aws/vendedlogs/states/waiter-state-machine-",
{
"Ref": "ProviderframeworkisComplete26D7B0CB"
},
"-c8100e0c94c82f842732ba88b108b9eee575b910e3"
]
]
},
"RetentionInDays": 731,
"Tags": [
{
"Key": "CmBillingGroup",
"Value": "CRM Shared"
}
]
},
"UpdateReplacePolicy": "Retain",
"DeletionPolicy": "Retain",
"Metadata": {
"aws:cdk:path": "Kubernetes/@aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine/LogGroup/Resource"
}
},
```

### Possible Solution

_No response_

### Additional Information/Context

I saw people say that the resource policies for logs have a limited size but I am not sure if that is the issue as the logs are in `/aws/vendedlogs/states/`:
```bash
aws logs describe-resource-policies | wc
54 85 4575
```

### CDK CLI Version

2.178.1 (build ae342cb)

### Framework Version

_No response_

### Node.js Version

v22.11.0

### OS

macOS Version 15.2 (24C101)

### Language

TypeScript

### Language Version

5.6.3

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with aws-cdk-lib 2.178.1 and inspect the generated template for @aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine, especially its IAM policy and LogGroup. Trace how the waiter state machine configures its log destination and verify the generated permissions against the AccessDenied error; done means the EKS cluster provider creates successfully without the Step Functions log authorization failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, typescript
Domain
cloud, infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.