aws / aws/aws-cdk

(eks): EKS cluster is created but the stack times out

Open
#15,608 8 comments 5 reactions 0 assignees View on GitHub
@aws-cdk/aws-eks bug effort/medium p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
1d 19h
Merged PRs (30d)
71

Description

We're experiencing an issue where an EKS cluster is deployed, the `ProviderframeworkisComplete` lambda will report back `SUCCESS`, but the CF stack will not move further along and create the node group we've specified. It's as if CF doesn't get the `SUCCESS` response. The stack will eventually timeout and rollback leading to another issue where the `OnEventHandler` reports an error that it cannot delete the cluster.

Perhaps there's something simple I've missed but I have yet to see what the error is here.

### Reproduction Steps

#### VPC:
```ts
const vpc = new ec2.Vpc(this, 'VPC', {
subnetConfiguration: [
{
cidrMask: 24,
name: 'Ingress',
subnetType: ec2.SubnetType.PUBLIC,
},
{
cidrMask: 24,
name: 'Application',
subnetType: ec2.SubnetType.PRIVATE,
},
{
cidrMask: 28,
name: 'Database',
subnetType: ec2.SubnetType.ISOLATED,
}
]
});
```

#### Cluster:

```ts
const mastersRole = new iam.Role(this, 'MastersRole', {
assumedBy: new iam.AccountRootPrincipal(),
});

// The IAM role that will be used by EKS
const role = new iam.Role(this, 'ClusterRole', {
assumedBy: new iam.ServicePrincipal('eks.amazonaws.com'),
managedPolicies: [
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKSClusterPolicy'),
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKSServicePolicy'),
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKSVPCResourceController'), // NOTE: Required for Security Groups for pods
],
});

// The EKS cluster, without worker nodes as we'll add them later
const cluster = new eks.Cluster(this, 'SimplyECluster', {
clusterName: `${product}-${environment}-cluster`,
defaultCapacity: 0,
mastersRole,
outputClusterName: true,
outputConfigCommand: true,
outputMastersRoleArn: true,
placeClusterHandlerInVpc: true,
role,
version: eks.KubernetesVersion.V1_20,
vpc,
});
cluster.node.addDependency(mastersRole);
cluster.node.addDependency(role);

// Managed Worker Nodes
//

// Worker node IAM role
const nodeRole = new iam.Role(this, 'NodeRole', {
assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
managedPolicies: [
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEC2ContainerRegistryReadOnly'),
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKSVPCResourceController'), // Allows us to use Security Groups for pods
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKSWorkerNodePolicy'),
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKS_CNI_Policy'),
iam.ManagedPolicy.fromAwsManagedPolicyName('ElasticLoadBalancingFullAccess'),
],
});

const nodeGroup = cluster.addNodegroupCapacity('NodeGroup', {
subnets: vpc.selectSubnets({ subnetType: ec2.SubnetType.PRIVATE, }),
nodeRole,
maxSize: 20,
minSize: 3,
});
nodeGroup.node.addDependency(nodeRole);
```

### What did you expect to happen?

EKS cluster creation.

### What actually happened?

Timeout and rollback of the stack even though the cluster was created.

### Environment

- **CDK CLI Version :** 7.18.1
- **Framework Version:** v1.114.0
- **Node.js Version:** v16.4.1
- **Language (Version):** TypeScript (3.9.10)

### Other

#### OnEventHandler cluster delete log

```
"2021-07-16T16":"23":45.692Z 7654cee8-a8e2-4072-b62f-6498e4498a5e ERROR Invoke Error{
"errorType":"AccessDeniedException",
"errorMessage":"User: arn:aws:sts::535241886961:assumed-role/simplye-dev-infra-Cluster-SimplyEClusterCreationRo-1EXTOZ12F2O0U/AWSCDK.EKSCluster.Delete.33864f3f-924a-493f-bd0f-06392c788668 is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:us-east-2:535241886961:cluster/simplye-dev-infra-ClusterNestedStackClusterNestedStackResourceC524F2E7-1P3CRQCDZN4ZI-SimplyEClusterC241F8AE-1XO6PW9I5RD9U",
"code":"AccessDeniedException",
"message":"User: arn:aws:sts::535241886961:assumed-role/simplye-dev-infra-Cluster-SimplyEClusterCreationRo-1EXTOZ12F2O0U/AWSCDK.EKSCluster.Delete.33864f3f-924a-493f-bd0f-06392c788668 is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:us-east-2:535241886961:cluster/simplye-dev-infra-ClusterNestedStackClusterNestedStackResourceC524F2E7-1P3CRQCDZN4ZI-SimplyEClusterC241F8AE-1XO6PW9I5RD9U",
"time":"2021-07-16T16:23:45.673Z",
"requestId":"4042285d-156e-4824-8f51-b29f5a67858a",
"statusCode":403,
"retryable":false,
"retryDelay":3.510279726269361,
"stack":[
"AccessDeniedException: User: arn:aws:sts::535241886961:assumed-role/simplye-dev-infra-Cluster-SimplyEClusterCreationRo-1EXTOZ12F2O0U/AWSCDK.EKSCluster.Delete.33864f3f-924a-493f-bd0f-06392c788668 is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:us-east-2:535241886961:cluster/simplye-dev-infra-ClusterNestedStackClusterNestedStackResourceC524F2E7-1P3CRQCDZN4ZI-SimplyEClusterC241F8AE-1XO6PW9I5RD9U",
" at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)",
" at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)",
" at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
" at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)",
" at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:688:14)",
" at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)",
" at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)",
" at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10",
" at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)",
" at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:690:12)"
]
}
```

---

This is :bug: Bug Report

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied TypeScript EKS configuration and inspect the ProviderframeworkisComplete callback and OnEventHandler cleanup logs. Trace whether CloudFormation receives the reported SUCCESS response and examine the shown eks:DeleteCluster AccessDeniedException. Done means the stack advances to create the node group and rollback can clean up the cluster without the reported errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.