aws / aws/containers-roadmap

CoreDNS causing an issue to InsufficientNumberOfReplicas - The add-on is unhealthy because it doesn't have the desired number of replicas.[service] [request]: describe request here

Open
#2,328 2 comments 4 reactions 0 assignees View on GitHub
EKS EKS Add-Ons EKS Networking Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### **Problem Description**

CoreDNS causing an issue to InsufficientNumberOfReplicas - The add-on is unhealthy because it doesn't have the desired number of replicas.

```
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "~> 20.0"

cluster_name = local.name
cluster_version = local.cluster_version

cluster_endpoint_private_access = true
cluster_endpoint_public_access = true

enable_irsa = true

enable_cluster_creator_admin_permissions = true

vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets

eks_managed_node_groups = {
cis_ami = {
instance_types = ["m5.large"]

ami_id = data.aws_ami.image.id

# # This will ensure the bootstrap user data is used to join the node
enable_bootstrap_user_data = true

iam_role_attach_cni_policy = true

min_size = 1
max_size = 6
desired_size = 4

}
}

# EKS Addons
cluster_addons = {
coredns = {
most_recent = true
}
kube-proxy = {
most_recent = true
}
# aws-ebs-csi-driver = {
# service_account_role_arn = module.ebs_csi_driver_irsa.iam_role_arn
# }
vpc-cni = {

before_compute = true
most_recent = true
configuration_values = jsonencode({
env = {
ENABLE_PREFIX_DELEGATION = "true"
WARM_PREFIX_TARGET = "1"
}
})
}
}

tags = local.tags
}
```

**Security disclosures**

I noticed that CoreDNS pods are crashing specifically when using the CIS EKS-Optimized Amazon Linux 2 Level 1 Image with a custom AMI."

```
❯ k get po -n kube-system
NAME READY STATUS RESTARTS AGE
aws-load-balancer-controller-54f58989fd-hj848 0/1 CrashLoopBackOff 17 (3m49s ago) 70m
aws-load-balancer-controller-54f58989fd-k2qzn 0/1 CrashLoopBackOff 17 (4m1s ago) 70m
aws-node-9bkt8 2/2 Running 0 71m
aws-node-psdvq 2/2 Running 0 71m
aws-node-qmhxg 2/2 Running 0 71m
aws-node-xl99d 2/2 Running 0 71m
cluster-autoscaler-aws-cluster-autoscaler-848fbf899c-8nxls 0/1 CrashLoopBackOff 16 (90s ago) 66m
coredns-557586b4b9-hnlg5 0/1 Running 0 64m
coredns-6f99ddbc54-pkltm 0/1 Running 0 56m
coredns-6f99ddbc54-xw65l 0/1 Running 0 56m
ebs-csi-controller-576c8d5c58-4q6vc 6/6 Running 0 69m
ebs-csi-controller-576c8d5c58-qk6m9 6/6 Running 0 70m
ebs-csi-node-5fztg 1/3 CrashLoopBackOff 40 (3m12s ago) 71m
ebs-csi-node-7tnrt 1/3 CrashLoopBackOff 41 (2m47s ago) 71m
ebs-csi-node-bmpqh 2/3 CrashLoopBackOff 41 (3m1s ago) 71m
ebs-csi-node-splqt 1/3 CrashLoopBackOff 39 (3m48s ago) 71m
kube-proxy-76gkn 1/1 Running 0 71m
kube-proxy-gkhcn 1/1 Running 0 71m
kube-proxy-hqxw7 1/1 Running 0 71m
kube-proxy-kxfds 1/1 Running

```
-->

**Which service(s) is this request for?**
EKS

**Attachments**

![Snip20240409_29](https://github.com/aws/containers-roadmap/assets/38572056/0aaec0ea-bc70-4c27-bbf1-ca065fcb2226)
![Snip20240409_30](https://github.com/aws/containers-roadmap/assets/38572056/b08fc7ce-da54-45f7-a1cd-747f25fa1171)

Contributor guide

Open the contributing guide

Research direction

Start from the Terraform EKS module configuration in the report and reproduce the cluster state with the shown kubectl get po -n kube-system command. Inspect the CoreDNS add-on and crashing pods in the CIS EKS-Optimized Amazon Linux 2 custom-AMI setup; done requires identifying the cause of the unhealthy replicas and documenting a verified resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, terraform
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
22/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.