aws / aws/aws-cdk

aws-eks: Cluster rollback fails all future deployments

Open
#31,626 4 comments 0 reactions 1 assignee Claimed by @moelasmar View on GitHub
@aws-cdk/aws-eks bug effort/medium p1
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
1d 19h
Merged PRs (30d)
71

Description

### Describe the bug

Cluster rollbacks can persistently break the cloudformation stack.

The issue occurs when 1) A cluster re-creation is triggered and 2) The deployment rolls back after the new cluster is created. When rolling back to the original cluster, parameters from the new (but now deleted) cluster are retained and fail all future deployments even if the cluster recreation commit is rolled back.

Essentially, while rolling back the stack, it needs to also roll back cached cluster information back to the original cluster's details.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Last Known Working CDK Version

_No response_

### Expected Behavior

Rollbacks should leave the stack in a functional state, reverting or fixing the CDK should allow new deployments to succeed.

### Current Behavior

Rollbacks leave the stack in a dysfunctional state, reverting the CDK code still results in failed stacks.

### Reproduction Steps

```
import { KubectlV24Layer } from '@aws-cdk/lambda-layer-kubectl-v24';
import { Cluster, KubernetesVersion } from 'aws-cdk-lib/aws-eks';
import { Construct } from 'constructs';
```

```
const addBreakingChange = false;

const cluster = new Cluster(this, 'MyCluster', {
kubectlLayer: new KubectlV24Layer(this, 'KubectlLayer'),
version: KubernetesVersion.V1_24,
defaultCapacity: 0,
clusterName: addBreakingChange ? 'newcluster' : undefined,
});

if (addBreakingChange) {
cluster.addManifest('eks-sample-linux-service', {
apiVersion: 'v1',
kind: 'Service',
metadata: {
name: 'eks-sample-linux-service',
namespace: 'eks-sample-app',
labels: {
app: 'non-existent-app',
},
},
});
}
```

Deployment 1: Create the cluster
Deployment 2: Change `addBreakingChange` to `true`, deployment fails
Deployment 3: Revert `addBreakingChange` to `false`, deployments will still fail

#### Without any additional complications, the failure message is
```
Resource handler returned message: "Error occurred while DescribeSecurityGroups. EC2 Error Code: InvalidGroup.NotFound. EC2 Error Message: The security group 'sg-[...]' does not exist (Service: Lambda, Status Code: 400, Request ID: [...])" (RequestToken: [...], HandlerErrorCode: InvalidRequest)
```

### Possible Solution

Workaround: I have found out that adding a tag to the cluster successfully triggers IsComplete to update the cluster parameters, e.g. security group.

### Additional Information/Context

Support Case ID: 172677804000994

### CDK CLI Version

2.113.0 (build ccd534a)

### Framework Version

_No response_

### Node.js Version

18

### OS

Amazon Linux 2 x86_64

### Language

TypeScript

### Language Version

TypeScript (5.0.4)

### Other information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.