(eks): EKS cluster is created but the stack times out
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 71
Description
We're experiencing an issue where an EKS cluster is deployed, the `ProviderframeworkisComplete` lambda will report back `SUCCESS`, but the CF stack will not move further along and create the node group we've specified. It's as if CF doesn't get the `SUCCESS` response. The stack will eventually timeout and rollback leading to another issue where the `OnEventHandler` reports an error that it cannot delete the cluster.
Perhaps there's something simple I've missed but I have yet to see what the error is here.
### Reproduction Steps
#### VPC:
```ts
const vpc = new ec2.Vpc(this, 'VPC', {
subnetConfiguration: [
{
cidrMask: 24,
name: 'Ingress',
subnetType: ec2.SubnetType.PUBLIC,
},
{
cidrMask: 24,
name: 'Application',
subnetType: ec2.SubnetType.PRIVATE,
},
{
cidrMask: 28,
name: 'Database',
subnetType: ec2.SubnetType.ISOLATED,
}
]
});
```
#### Cluster:
```ts
const mastersRole = new iam.Role(this, 'MastersRole', {
assumedBy: new iam.AccountRootPrincipal(),
});
// The IAM role that will be used by EKS
const role = new iam.Role(this, 'ClusterRole', {
assumedBy: new iam.ServicePrincipal('eks.amazonaws.com'),
managedPolicies: [
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKSClusterPolicy'),
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKSServicePolicy'),
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKSVPCResourceController'), // NOTE: Required for Security Groups for pods
],
});
// The EKS cluster, without worker nodes as we'll add them later
const cluster = new eks.Cluster(this, 'SimplyECluster', {
clusterName: `${product}-${environment}-cluster`,
defaultCapacity: 0,
mastersRole,
outputClusterName: true,
outputConfigCommand: true,
outputMastersRoleArn: true,
placeClusterHandlerInVpc: true,
role,
version: eks.KubernetesVersion.V1_20,
vpc,
});
cluster.node.addDependency(mastersRole);
cluster.node.addDependency(role);
// Managed Worker Nodes
//
// Worker node IAM role
const nodeRole = new iam.Role(this, 'NodeRole', {
assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
managedPolicies: [
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEC2ContainerRegistryReadOnly'),
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKSVPCResourceController'), // Allows us to use Security Groups for pods
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKSWorkerNodePolicy'),
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKS_CNI_Policy'),
iam.ManagedPolicy.fromAwsManagedPolicyName('ElasticLoadBalancingFullAccess'),
],
});
const nodeGroup = cluster.addNodegroupCapacity('NodeGroup', {
subnets: vpc.selectSubnets({ subnetType: ec2.SubnetType.PRIVATE, }),
nodeRole,
maxSize: 20,
minSize: 3,
});
nodeGroup.node.addDependency(nodeRole);
```
### What did you expect to happen?
EKS cluster creation.
### What actually happened?
Timeout and rollback of the stack even though the cluster was created.
### Environment
- **CDK CLI Version :** 7.18.1
- **Framework Version:** v1.114.0
- **Node.js Version:** v16.4.1
- **Language (Version):** TypeScript (3.9.10)
### Other
#### OnEventHandler cluster delete log
```
"2021-07-16T16":"23":45.692Z 7654cee8-a8e2-4072-b62f-6498e4498a5e ERROR Invoke Error{
"errorType":"AccessDeniedException",
"errorMessage":"User: arn:aws:sts::535241886961:assumed-role/simplye-dev-infra-Cluster-SimplyEClusterCreationRo-1EXTOZ12F2O0U/AWSCDK.EKSCluster.Delete.33864f3f-924a-493f-bd0f-06392c788668 is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:us-east-2:535241886961:cluster/simplye-dev-infra-ClusterNestedStackClusterNestedStackResourceC524F2E7-1P3CRQCDZN4ZI-SimplyEClusterC241F8AE-1XO6PW9I5RD9U",
"code":"AccessDeniedException",
"message":"User: arn:aws:sts::535241886961:assumed-role/simplye-dev-infra-Cluster-SimplyEClusterCreationRo-1EXTOZ12F2O0U/AWSCDK.EKSCluster.Delete.33864f3f-924a-493f-bd0f-06392c788668 is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:us-east-2:535241886961:cluster/simplye-dev-infra-ClusterNestedStackClusterNestedStackResourceC524F2E7-1P3CRQCDZN4ZI-SimplyEClusterC241F8AE-1XO6PW9I5RD9U",
"time":"2021-07-16T16:23:45.673Z",
"requestId":"4042285d-156e-4824-8f51-b29f5a67858a",
"statusCode":403,
"retryable":false,
"retryDelay":3.510279726269361,
"stack":[
"AccessDeniedException: User: arn:aws:sts::535241886961:assumed-role/simplye-dev-infra-Cluster-SimplyEClusterCreationRo-1EXTOZ12F2O0U/AWSCDK.EKSCluster.Delete.33864f3f-924a-493f-bd0f-06392c788668 is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:us-east-2:535241886961:cluster/simplye-dev-infra-ClusterNestedStackClusterNestedStackResourceC524F2E7-1P3CRQCDZN4ZI-SimplyEClusterC241F8AE-1XO6PW9I5RD9U",
" at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)",
" at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)",
" at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
" at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)",
" at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:688:14)",
" at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)",
" at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)",
" at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10",
" at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)",
" at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:690:12)"
]
}
```
---
This is :bug: Bug Report
Contributor guide
Research direction
Start by reproducing the supplied TypeScript EKS configuration and inspect the ProviderframeworkisComplete callback and OnEventHandler cleanup logs. Trace whether CloudFormation receives the reported SUCCESS response and examine the shown eks:DeleteCluster AccessDeniedException. Done means the stack advances to create the node group and rollback can clean up the cluster without the reported errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100