aws / aws/aws-cdk

aws-eks: Updating KubernetesManifest deletes it instead

Open
#33,406 3 comments 1 reaction 0 assignees View on GitHub
@aws-cdk/aws-eks @aws-cdk/aws-eks-v2-alpha bug effort/medium p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

Updating a KubernetesManifest resource through CDK can actually cause it to get deleted.

During a resource replacement, if `overwrite: true` and the previous manifest has any overlap with the new manifest, the overlapping section would be lost. When the manifest is unchanged, the entire resource is deleted. Issue cannot be mitigated by a rollback or code revert and will repeat on any subsequent update.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Last Known Working CDK Version

_No response_

### Expected Behavior

Replacing a KubernetesManifest should at most delete and re-create the underlying EKS manifest resources. A minimal update to a KubernetesManifest should not result in a loss of cluster functionality resulting from missing Kubernetes resources.

### Current Behavior

Updates to a KubernetesManifest which are applied as a replacement cause cluster resources to be wiped. Rollbacks and reverts do not bring the cluster back to a healthy state.

Given Manifest A (previous) and Manifest B (new) are based on the same yaml, replacing the KubernetesManifest resource looks like this:
1. Cloudformation first applies Manifest B, which overwrites Manifest A in EKS; nothing happens basically
2. Manifest A and B both exist in the cloudformation stack, manifest contents are correctly configured in EKS
3. Cloudformation deletes Manifest A, which deletes the manifest resources from EKS
4. Cloudformation now has "updated" to Manifest B, but nothing is in EKS anymore

### Reproduction Steps

### Setup
```ts
new eks.KubernetesManifest(cluster, 'Sleeper', {
manifest: [
{
apiVersion: 'v1',
kind: 'Pod',
metadata: {
name: 'test-sleeper',
},
spec: {
containers: [
{
name: 'sleeper',
image: 'alpine:latest',
imagePullPolicy: 'Always',
command: ['/bin/sleep', 'infinity'],
},
],
},
},
],
cluster,
overwrite: true,
});
```
```
> kubectl get pods

NAME READY STATUS RESTARTS AGE
test-sleeper 1/1 Running 0 40s
```

### Minimal Change
```diff
- new eks.KubernetesManifest(cluster, 'Sleeper', {
+ new eks.KubernetesManifest(cluster, 'Sleeper1', {
```

CloudFormation Events

| Timestamp | Logical ID | Status |
| :-: | :-: | :-: |
| 2025-02-11 13:49:18 UTC-0800 | ClusterSleeper0E1728F7 | DELETE_COMPLETE |
| 2025-02-11 13:48:38 UTC-0800 | ClusterSleeper0E1728F7 | DELETE_IN_PROGRESS |
| 2025-02-11 13:48:37 UTC-0800 | `` | UPDATE_COMPLETE_CLEANUP_IN_PROGRESS |
| 2025-02-11 13:48:17 UTC-0800 | ClusterSleeper1A9127B4A | CREATE_COMPLETE |
| 2025-02-11 13:48:17 UTC-0800 | ClusterSleeper1A9127B4A | CREATE_IN_PROGRESS (Resource creation Initiated) |
| 2025-02-11 13:48:05 UTC-0800 | ClusterSleeper1A9127B4A | CREATE_IN_PROGRESS |

```
> kubectl get pods

No resources found in default namespace.
```

### Reverts Are Ineffective
```diff
- new eks.KubernetesManifest(cluster, 'Sleeper1', {
+ new eks.KubernetesManifest(cluster, 'Sleeper', {
```

Similar events to above, sleeper pod is created then deleted again.

```
> kubectl get pods

No resources found in default namespace.
```

### Possible Solution

Immediate Mitigating Options:
- Trigger a minimal replacement and set the manifest's deletion policy to RETAIN
```ts
(manifest.node.defaultChild as CfnResource).applyRemovalPolicy(RemovalPolicy.RETAIN);
```
- Remove the manifest from CDK entirely, deploy, then add it back

Note: Using `RemovalPolicy.RETAIN` comes with the natural downside of having to clean up dangling resources manually

### Additional Information/Context

Additional Risks:

If we update manifests and there is any overlap between the original and subsequent manifests, CloudFormation might invisibly delete parts of a manifest. For example, if manifest version 1.0 is deployed and replaced with manifest version 2.0, the intersecting resources (1.0 ∩ 2.0) will be deleted when cleaning up 1.0.

### CDK CLI Version

2.160.0

### Framework Version

_No response_

### Node.js Version

18

### OS

Amazon Linux 2 x86_64

### Language

TypeScript

### Language Version

5.0.4

### Other information

Sev2: P199049085
Tracking: P200043360
Case ID 173931643600782

Contributor guide

Open the contributing guide

Research direction

Start with the KubernetesManifest resource replacement path and compare the CloudFormation create and cleanup sequence described in the issue. Reproduce the Sleeper-to-Sleeper1 change with CDK CLI 2.160.0 and verify that replacing a manifest leaves the underlying EKS resources intact, including when manifests overlap.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.