High cpu consumption of containerd and kubelet

Environment Details:

Cluster Setup: Production cluster with Cluster Autoscaler and Managed Node Groups.
OS/Images: AWS Bottlerocket AMIs (the issue persists across multiple Bottlerocket versions).
EKS Versions: 1.34 and 1.35.

Issue Description:
Starting a few weeks ago, we began experiencing node failures. The applications running on the affected nodes appear to be suffering from CPU throttling, and the node load averages are extremely high.

Investigation & Findings:

Metric Discrepancy: Despite the high node load averages, our pod CPU consumption metrics show only very modest usage.
High Component CPU: Upon further investigation using the admin container, the top command revealed that containerd and kubelet are consuming an unusually high amount of CPU. This explains the discrepancy between the high node load averages and the low pod CPU usage.
Log Analysis: The logs indicate that the kubelet is consistently asking containerd for status updates on containers. However, containerd is responding that it does not know the pods or cannot find the status/info for them. We suspect this continuous polling loop is the reason for the high CPU consumption in these two components.

Attachments Included:
To support these findings, I have attached the following evidence:
Screenshot showing the high node load averages.

Screenshot showing the modest CPU consumption per pod.

Screenshot of the top command output confirming the high CPU consumption of containerd and kubelet.