aws / aws/aws-cdk

[aws-eks] Stacks with EKS clusters fail tear down due to dangling ELB/ALB holding networking resources

Open
#9,970 20 comments 9 reactions 0 assignees View on GitHub
@aws-cdk/aws-eks bug effort/medium p1
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

## :question: General Issue

The tear down process of a stack runs into race conditions when Kubernetes Operators and Controllers are involved that manages external deployment of resources. As an example, say we have an ALB Ingress Controller deployed through a Helm chart. Following that, we deploy a couple Ingress resources which the ALB Ingress Controller will create ALBs for. When the stack is removed, the removal of k8s resources often orphans that cloud resource equivalents, which causes the stack to fail cleanup because these orphaned resources leave bread crumbs that blocks things like SG/VPC/ENI removals. Is there a way to clean up resources properly when using CDK with operator and controllers? I've tried separating K8s resources (helmchart/manifest) into a separate stack so I could manually invoke a `sleep` between the `cdk destroy` command. But ran into trouble even separating the stacks out due to the circular dependencies between the cluster and these K8s overlay resources.

A side note. I'm aware that ALB Ingress Controller introduced finalizers into the Ingress resource it manages. This means the resource doesn't delete from the K8s control plane until the Ingress Controller has removed all AWS resources which is good. Maybe the `aws-eks` resource delete mechanism doesn't wait?

These are some scenarios that I've found which causes race conditions during Stack teardown.

* ALB Ingress Controller
* Removal of Ingress object causes race condition between VPC deletion and ALB where ALB is not fully removed by the time VPC is being removed as well. Often I see ENI breadcrumbs that fails the VPC removal.
* Removal of ALB Ingress Controller before the Ingress object removal have been fulfilled in processing. This shows itself with a fully intact ALB which causes VPC deletion failures.
* External DNS Controller
* The External DNS Controller might be deleted before it has had time to process fulfillment for proxy resources that were decorated with dns entries. I often see Route53 zone with left over CNAME/A/TXT records owned by External DNS Controller.
* Any Operator/Deployment
* When resources are removed all at once, the kubelets have not had enough time to delete the supporting pods for the resource sets. But because acknowledgement from the K8s control plane has been sent that deletion of those resources is recorded, Cloudformation acknowledges as a fulfilled state. This often lets Cloudformation follow up with deleting of the managed worker pools. When this happens, the pods that weren't deleted leaves orphaned ENI since the EC2 supporting them no longer exist. This is possible when using the CNI that supports AWS VPC native networking.

Contributor guide

Open the contributing guide

Research direction

Start with the aws-eks resource delete mechanism and the cdk destroy flow, then review the Helm chart/manifest teardown scenarios described for ALB Ingress Controller, External DNS, and other operators. Done would require a defined, tested cleanup ordering or wait strategy that prevents dangling ALB, Route53, ENI, VPC, and security-group resources during stack teardown.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, helm, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.