[EKS Auto Mode] System reservations not being respected

Over the past month or two, we've had multiple occurrences of nodes having networking issues in our clusters. We're seeing nodes becoming unready when a pod CPU usage spikes, ready checks inconsistently failing, random pod restarts (primarily due to failed ready checks even when the pods are functioning normally), random network failures, etc.

AWS Business Support have told us on multiple occasions that this is because we have not set CPU limits on our pods, however we have set requests, and since Kubernetes considers CPU a compressible resource, we'd expect (and all the guidance tells us) that these nodes should be fine. In this case, we'd expect the host and networking on the node to have enough CPU reserved within the kubelet configuration that should not be impacted by pods consuming all of the space that has been allocated to pods. There are resources reserved within the kubelet config.

When these nodes died, they had hit 100% CPU used (Cloudwatch metrics, measured by AWS, presumably at the host?). Surely if there are reservations, this should never happen? The pods can't use the system reserved CPU?

Our existing clusters running the exact same workloads (we're migrating from EKS to EKS Auto) don't experience this issue (they run Cilium as a network fabric, it has some reservations?).

Can some clarification be provided on whether Auto Mode should be supporting CPU requests without limits, and if it should be, why it's not working? Is there anything we can do to debug this, or ourselves allocate more resources to the node to stop core processes being starved of resource?

No files, tests, or entry points are named. Start by reviewing the reported EKS Auto Mode nodes, kubelet system reservations, CPU requests without limits, and the CloudWatch CPU observations. Done means establishing whether the reservations are applied and whether the reported node failures can be explained or reproduced.

Written by the indexing model from the issue text.