aws / aws/aws-cdk

EKS: Cluster Deletion Fails

Open
#32,395 5 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-eks bug effort/large p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

A CFN stack containing an EKS cluster failed and attempted to roll back. The OnEventHandler custom resource that's responsible for handling cluster deletion failed to delete the resource with permissions error. From the CW logs:

```
2024-12-05T07:12:25.344Z df9e8909-0564-4c9e-9529-5546180edcec ERROR {
clientName: 'EKSClient',
commandName: 'DeleteClusterCommand',
input: {
name: 'multi-az-workshop-EKSNestedStackEKSNestedStackResourceAE427C53-2PG5GFQGAIDA-ClusterEKSClusterEAC9DE5C-N7TGM3G9041D'
},
error: AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/multi-az-workshop-EKSNest-ClusterEKSClusterCreation-2nnnu7xJhiuj/AWSCDK.EKSCluster.Delete.9e794145-daf9-44f3-88f2-d0cd7c694239 is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:us-west-2:123456789012:cluster/multi-az-workshop-EKSNestedStackEKSNestedStackResourceAE427C53-2PG5GFQGAIDA-ClusterEKSClusterEAC9DE5C-N7TGM3G9041D
at de_AccessDeniedExceptionRes (/var/runtime/node_modules/@aws-sdk/client-eks/dist-cjs/index.js:2546:21)
at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-eks/dist-cjs/index.js:2519:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:165:18
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
at async Xi.onDelete (/var/task/index.js:57:649490) {
'$fault': 'client',
'$metadata': {
httpStatusCode: 403,
requestId: '68a18a1d-2119-42f0-aeae-9253766fdf5a',
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
}
},
metadata: {
httpStatusCode: 403,
requestId: '68a18a1d-2119-42f0-aeae-9253766fdf5a',
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
}
}
```

However, this doesn't happen all of the time, and am wondering if there is a hidden race condition during a stack rollback where the permissions policy may get deleted before the function and assigned role are deleted?

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Last Known Working CDK Version

_No response_

### Expected Behavior

I would expect the automatically created custom resource and IAM role to have the appropriate permissions.

### Current Behavior

Sometimes, cluster deletion fails with a 403 error.

### Reproduction Steps

I don't have specific reproduction steps since the behavior is transient. This is basically the cluster resource definition:

```TypeScript
Cluster cluster = new Cluster(this, "EKSCluster", new ClusterProps(){
Vpc = props.Vpc,
VpcSubnets = new SubnetSelection[] { new SubnetSelection() { SubnetType = SubnetType.PRIVATE_ISOLATED } },
DefaultCapacity = 0,
Version = KubernetesVersion.V1_31,
PlaceClusterHandlerInVpc = false,
EndpointAccess = EndpointAccess.PUBLIC_AND_PRIVATE,
KubectlLayer = kubetctlLayer,
SecurityGroup = controlPlaneSG,
MastersRole = props.AdminRole,
ClusterName = props.ClusterName,
ClusterLogging = new ClusterLoggingTypes[] { ClusterLoggingTypes.CONTROLLER_MANAGER, ClusterLoggingTypes.AUTHENTICATOR, ClusterLoggingTypes.API, ClusterLoggingTypes.AUDIT, ClusterLoggingTypes.SCHEDULER}
});
```

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### CDK CLI Version

2.164.1

### Framework Version

_No response_

### Node.js Version

20

### OS

darwin

### Language

.NET

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing the OnEventHandler custom resource's EKS DeleteClusterCommand path and the automatically created IAM role and permissions policy during CloudFormation rollback. Use the provided AccessDeniedException and cluster definition to investigate whether deletion ordering can remove permissions too early. Done means transient rollback deletion no longer produces the reported 403 error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.