No Route to Host Errors on app while nodes are scaling down in AWS EKS

Hi,

I have a AWS EKS 1.21 cluster that all my applications deployed and working fine in a normal circumstances. I have active HPAs for my apps and cluster-autoscaler for my node autoscaling. My issue is about closing nodes. When a node is no longer needed and cluster starts to closing it, I'm seeing No route to host errors in my application logs for a short amount of time. I'm suspecting **CoreDNS cache**. Although pods/nodes are being closed, is there a possibility that CoreDNS is still trying to send traffic to closed pods/nodes? This is first possibility that comes to my mind and **I couldn't change CoreDNS configmap to disable cache because it's being reverted all the time.**

I neither identified the problem nor came with any other ideas.

Additional information about the system and the deployments;

**CoreDNS addon version** : `coredns:v1.8.4-eksbuild.1`
Addons created via` aws_eks_addon` resource in Terraform.

There is no extra DNS configuration on deployment files, so `DNSPolicy` is working as `ClusterFirst`
My `ndot` configuration is default which is `ndots:5`

**My CoreDNS configmap :**

```
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
labels:
eks.amazonaws.com/component: coredns
k8s-app: kube-dns
name: coredns
namespace: kube-system
```

**My cluster autoscaler configuration;**

```
spec:
containers:
- command:
- ./cluster-autoscaler
- --v=4
- --stderrthreshold=info
- --cloud-provider=aws
- --skip-nodes-with-local-storage=false
- --expander=random
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/xxxx
- --scan-interval=10s
- --max-empty-bulk-delete=1
- --scale-down-unneeded-time=5m
- --scale-down-utilization-threshold=0.60
- --scale-down-delay-after-add=5m
- --scale-down-delay-after-delete=5m
- --max-graceful-termination-sec=180
image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.21.2
imagePullPolicy: Always
name: cluster-autoscaler
resources:
limits:
cpu: 100m
memory: 300Mi
requests:
cpu: 100m
memory: 300Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/ssl/certs/ca-certificates.crt
name: ssl-certs
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: cluster-autoscaler
serviceAccountName: cluster-autoscaler
terminationGracePeriodSeconds: 30
```

Any fresh ideas could be nice here, I'm really stuck and I don't know what I'm missing.

Thank you very much, Best Regards.

Start with the CoreDNS ConfigMap and cluster-autoscaler deployment configuration included in the report, then correlate the application’s no-route errors with node scale-down and pod termination events. Done requires identifying whether DNS, service endpoints, or node termination causes the transient failures and documenting a reproducible mitigation.

Written by the indexing model from the issue text.