Unresponsive/unreachable Bottlerocket EKS nodes

Hey folks,

Coming to you with an odd issue with Bottlerocket EKS nodes becoming unresponsive and unreachable, but still happily running as seen by EC2.

**Image I'm using:**

`ami-089e696e7c541c61b` (amazon/bottlerocket-aws-k8s-1.29-aarch64-v1.20.2-536d69d0)

```
System Info:
Kernel Version: 6.1.90
OS Image: Bottlerocket OS 1.20.2 (aws-k8s-1.29)
Operating System: linux
Architecture: arm64
Container Runtime Version: containerd://1.6.31+bottlerocket
Kubelet Version: v1.29.1-eks-61c0bbb
Kube-Proxy Version: v1.29.1-eks-61c0bbb
```

We use Karpenter as a node provisioner.

**What I expected to happen:**

Node runs smoothly.

**What actually happened:**

A large node (`m7g.8xlarge`) is running a memory-heavy JVM workload from a StatefulSet that takes up most of the machine (request=115Gi out of 128Gi).

Node runs happily for a while (~ 24h to several days), until it suddenly stops reporting status to Kubernetes:
```
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure Unknown Tue, 25 Jun 2024 15:18:44 +0200 Tue, 25 Jun 2024 15:20:38 +0200 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Tue, 25 Jun 2024 15:18:44 +0200 Tue, 25 Jun 2024 15:20:38 +0200 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Tue, 25 Jun 2024 15:18:44 +0200 Tue, 25 Jun 2024 15:20:38 +0200 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Tue, 25 Jun 2024 15:18:44 +0200 Tue, 25 Jun 2024 15:20:38 +0200 NodeStatusUnknown Kubelet stopped posting node status.
```

Kubernetes marks that node as NotReady and taints it with `node.kubernetes.io/unreachable`, then tries to delete the pod running our large workload. That doesn't work (the node is fully unresponsive), so that pod is stuck in status Terminating.

The node is unreachable via `kubectl debug node ...` (which times out), or via AWS SSM (which complains of "SSM Agent is not online", "Ping status: Connection lost"). The EC2 Serial Console is empty (all black). We do not have SSH enabled on these machines.

However, the node still appears as running from the EC2 console, reachability metrics are green, and we can see CPU / network metrics flowing in.

I've managed to get system logs by taking an EBS snapshot of the Bottlerocket data volume and restoring it to a separate volume for investigation. This was not helpful unfortunately: logs appear normal until the time the node dies, then suddenly stop. There is no indication that anything in particular (kubelet, containerd, etc) crashed and brought the instance down, but also, suspisciously, no logs _at all_ from the moment the node went unresponsive.

**How to reproduce the problem:**

No clear way to reproduce unfortunately; we've seen this happen sporadically on maybe half a dozen instances over the past few weeks, out of several hundred

This issue is very annoying: I don't mind having pods crashing and/or getting evicted sometimes, or even kubelet/containerd crashing, but I'd expect it to self-heal eventually. This causes our workloads to get stuck, and we have to manually delete the pod and/or the node to get it back to normal. But even worse, I can't see a way to debug it properly or get to the bottom of it.

Would you have any idea of a better way to debug this?

Thank you!

(Note: I also opened an AWS support ticket to see if there's any AWS-level issue at play here, but this seems to happen only on this specific workload on Bottlerocket nodes, so I suspect something is off here)

The report names no source files, tests, or reproducible entry point. Start by reviewing the Bottlerocket system logs around the last kubelet heartbeat and compare them with EC2, SSM, and serial-console evidence. Done would require identifying a reproducible cause or a documented recovery path, but the issue currently provides neither.

Written by the indexing model from the issue text.