AKS sysbox nodes sporadically enter bad state (`stat failed on /var/lib/containers/storage/overlay/...`)

I'm attempting to port our CI kubernetes cluster to use sysbox to enable building and running docker containers inside our CI jobs.

We're finding that after a number of CI builds, nodes sporadically get into a bad state and are unable to start new containers (pods with status CreateContainerError). For pods/containers that are already running, their local docker daemon gets stuck and the CI build jobs running on them (mostly just docker builds) time out.

We are running roughly 14 pods per node, and we restart our pods (including their local dockerd process) after every job. The cluster is hosted by AKS on Azure. My sample size isn't enormous, but this issue seems to reproduce more frequently on sysbox 0.6.3 (after ~30 builds per node) vs 0.6.2 with SYSBOX_ALLOW_TRUSTED_XATTR set to FALSE (after ~120 builds per node), however it does reproduce on both. The node's kernel version is 5.15.

kubectl describe node doesn't show anything out of the ordinary, the node is ready and hosting pods (though they cannot start) and doesn't have any errors flagged on it.

journalctl -u kubelet does contain error log lines that healthy nodes do not contain, namely many lines that look like:

W0129 09:05:08.825295   17183 container.go:589] Failed to update stats for container "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753ba6ae_0b8f_47c8_992c_0f9eac91476a.slice/crio-761d1ec685b47ab5c8df26d44bb00c748610eb39c834c7d43383f749ff841835.scope": unable to determine device info for dir: /var/lib/containers/storage/overlay/2e13b53c75be18eb8c3f3773de0c861efe4135b1298157f5d981b3fea38c25ce/diff: stat failed on /var/lib/containers/storage/overlay/2e13b53c75be18eb8c3f3773de0c861efe4135b1298157f5d981b3fea38c25ce/diff with error: no such file or directory, continuing to push stats

journalctl -u crio also contains error log lines that healthy nodes do not have, but it seems just a result of the containers not being able to start:

time="2024-01-29 05:09:24.451212422Z" level=warning msg="Unable to delete container b01fadc160a9a10c4da6ac84c1d6e6082a30119de55eec23a3b9e855d71f6e04: `/usr/bin/sysbox-runc --root /run/runc --systemd-cgroup delete --force b01fadc160a9a10c4da6ac84c1d6e6082a30119de55eec23a3b9e855d71f6e04` failed: time=\"2024-01-29T05:09:24Z\" level=error msg=\"container init still running\"\n : exit status 1" id=3d1dc20d-694b-46b3-a7e4-4f38ffa6939c name=/runtime.v1.RuntimeService/CreateContainer

crictl stop $STUCK_CONTAINER_ID gives the following message:

E0129 09:10:39.944756  458151 remote_runtime.go:505] "StopContainer from runtime service failed" err="rpc error: code = DeadlineExceeded desc = failed to stop container 7f5d0d7dbffa70f5a34a07a62ed25975a1910596819e29803c2b75c413af48fc: context deadline exceeded" containerID="7f5d0d7dbffa7"
FATA[0002] stopping the container "7f5d0d7dbffa7": rpc error: code = DeadlineExceeded desc = failed to stop container 7f5d0d7dbffa70f5a34a07a62ed25975a1910596819e29803c2b75c413af48fc: context deadline exceeded

I don't see anything out of the ordinary in journalctl -u sysbox-mgr or journalctl -u sysbox-fs on the bad node.

Are there more places that I can find potentially helpful information here?

Start by comparing healthy and bad AKS nodes using the kubelet, CRI-O, sysbox-mgr, and sysbox-fs journal entries described in the issue. Reproduce the failure around repeated pod and Docker daemon restarts, then use the reported overlay path and crictl stop timeout to correlate the runtime errors. Done means identifying a supported cause or a focused set of diagnostics explaining why nodes stop starting containers.