K8S Pods with gVisor are stuck in running state with load of zombie children

### Description

We have a Kubernetes cluster on which we are observing the following behaviour for quite some time:

```
$ k get po
NAME READY STATUS RESTARTS AGE
4db95b31-ba4c-4de9-904e-391f7a0df5f5-1759791593-driver-0 1/1 Running 0 1d12h 2025-10-06T23:03:16Z ip-172-30-4-239.eu-central-1.compute.internal
```

That Pod claims to be running but it should have terminated quite some time ago. That Pod is running in a gVisor sandbox and contains a huge amount of zombie children. The Pods sandbox id is `ae0b66c097179b61e39a73400d574b4c19812d527d2e2ad9c4dfe92f767246ff`.

When we get on to the node, this is the process tree starting with the shim:
```
root 3999813 0.0 0.0 1236552 11352 ? Sl Oct06 0:07 /var/bin/containerruntimes/containerd-shim-runsc-v1 -namespace k8s.io -address /run/containerd/containerd.sock -publish-binary /usr/bin/containerd
root 3999846 0.0 0.0 1273180 26988 ? Ssl Oct06 0:02 \_ runsc-gofer --log-format=json --panic-log=/var/log/pods/c9ecbc0f-4819-4868-b337-c17e3e4dc9eb_4db95b31-ba4c-4de9-904e-391f7a0df5f5-1759791593-driver-0_b6935fdf-7909-4c24-9a4e-3f25567a0681/gvisor_pan
root 3999850 9.2 0.0 3596852 71420 ? Ssl Oct06 320:22 \_ runsc-sandbox --log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/ae0b66c097179b61e39a73400d574b4c19812d527d2e2ad9c4dfe92f767246ff/log.json --log-format=json --panic-log=/var/log/pods/c9ecbc
root 3999909 0.0 0.0 16456 0 ? Ss Oct06 0:00 | \_ [exe]
root 3999953 0.0 0.0 17992 0 ? S Oct06 0:00 | \_ [exe]
root 3999954 0.0 0.0 17992 0 ? SN Oct06 0:00 | | \_ [exe]
root 4000220 0.0 0.0 16672 28 ? S Oct06 0:00 | \_ [exe]
root 4000221 0.0 0.0 16672 28 ? SN Oct06 0:00 | | \_ [exe]
root 4001010 0.0 0.0 16672 28 ? SN Oct06 0:00 | | \_ [exe]
root 4001011 0.0 0.0 16672 28 ? SN Oct06 0:00 | | \_ [exe]
root 4001016 0.0 0.0 16672 28 ? SN Oct06 0:00 | | \_ [exe]
root 4001017 0.0 0.0 16672 28 ? SN Oct06 0:00 | | \_ [exe]
root 4001018 0.0 0.0 16672 28 ? SN Oct06 0:00 | | \_ [exe]
root 4001029 0.0 0.0 16672 28 ? SN Oct06 0:00 | | \_ [exe]
root 4001030 0.0 0.0 16672 28 ? SN Oct06 0:00 | | \_ [exe]
root 4001031 0.0 0.0 16672 28 ? SN Oct06 0:00 | | \_ [exe]
root 4000543 0.0 0.0 16624 564 ? S Oct06 0:00 | \_ [exe]
root 4000544 0.0 0.0 16624 564 ? SN Oct06 0:00 | | \_ [exe]
root 4000551 0.0 0.0 16624 564 ? SN Oct06 0:00 | | \_ [exe]
root 4000552 0.0 0.0 16624 564 ? SN Oct06 0:00 | | \_ [exe]
root 4000553 0.0 0.0 16624 564 ? SN Oct06 0:00 | | \_ [exe]
root 4000566 0.0 0.0 16624 564 ? SN Oct06 0:00 | | \_ [exe]
root 4000567 0.0 0.0 16624 564 ? SN Oct06 0:00 | | \_ [exe]
root 4000601 0.0 0.0 16624 564 ? SN Oct06 0:00 | | \_ [exe]
root 4000545 0.0 0.0 16480 768 ? S Oct06 0:00 | \_ [exe]
root 4000546 0.0 0.0 16480 768 ? SN Oct06 0:00 | | \_ [exe]
root 4001552 0.0 0.0 16480 0 ? S Oct06 0:00 | \_ [exe]
root 4001553 0.0 0.0 16480 0 ? SN Oct06 0:00 | | \_ [exe]
root 4001554 0.0 0.0 16480 0 ? S Oct06 0:00 | \_ [exe]
root 4001556 0.0 0.0 16480 0 ? SN Oct06 0:00 | | \_ [exe]
root 4001555 0.0 0.0 16936 0 ? S Oct06 0:00 | \_ [exe]
root 4001557 0.0 0.0 16936 0 ? SN Oct06 0:01 | | \_ [exe]
root 4001585 0.0 0.0 16936 0 ? SN Oct06 0:01 | | \_ [exe]
root 4001586 0.0 0.0 16936 0 ? SN Oct06 0:01 | | \_ [exe]
root 4001587 0.0 0.0 16936 0 ? SN Oct06 0:01 | | \_ [exe]
root 4001588 0.0 0.0 16936 0 ? SN Oct06 0:01 | | \_ [exe]
root 4001589 0.0 0.0 16936 0 ? SN Oct06 0:01 | | \_ [exe]
root 4001595 0.0 0.0 16936 0 ? SN Oct06 0:01 | | \_ [exe]
root 4001596 0.0 0.0 16936 0 ? SN Oct06 0:01 | | \_ [exe]
root 4001597 0.0 0.0 16936 0 ? SN Oct06 0:01 | | \_ [exe]
root 4001599 0.0 0.0 16936 0 ? SN Oct06 0:01 | | \_ [exe]
root 4001600 0.0 0.0 16936 0 ? SN Oct06 0:01 | | \_ [exe]
root 4001601 0.0 0.0 16936 0 ? SN Oct06 0:01 | | \_ [exe]
root 4001608 0.0 0.0 16936 0 ? SN Oct06 0:00 | | \_ [exe]
root 4001617 0.0 0.0 16936 0 ? SN Oct06 0:00 | | \_ [exe]
root 4001619 0.0 0.0 16936 0 ? SN Oct06 0:00 | | \_ [exe]
root 4001621 0.0 0.0 16936 0 ? SN Oct06 0:00 | | \_ [exe]
root 4001622 0.0 0.0 16936 0 ? SN Oct06 0:00 | | \_ [exe]
root 4001623 0.0 0.0 16936 0 ? SN Oct06 0:00 | | \_ [exe]
root 4001624 0.0 0.0 16936 0 ? SN Oct06 0:00 | | \_ [exe]
root 4001625 0.0 0.0 16936 0 ? SN Oct06 0:00 | | \_ [exe]
root 4001559 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001647 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001648 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001649 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001651 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001652 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001653 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001654 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001659 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001678 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001679 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001758 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001759 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001760 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001761 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001762 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001763 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001764 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001765 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001766 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001767 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001768 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001769 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001770 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001771 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001772 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001773 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001774 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001775 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001776 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001783 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001784 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001785 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001786 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001787 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001788 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001789 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001790 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001791 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001793 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001794 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001795 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 4001796 0.0 0.0 0 0 ? ZN Oct06 0:00 | \_ [exe]
root 3999955 0.0 0.0 1272028 22180 ? Sl Oct06 0:00 \_ runsc --root=/run/containerd/runsc/k8s.io --log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/ae0b66c097179b61e39a73400d574b4c19812d527d2e2ad9c4dfe92f767246ff/log.json --log-format=json --pa
```

If we would like to query runsc to list all containers, it blocks forever:

```
$ root@ip-172-30-4-239:/var/bin/containerruntimes#./runsc --root /run/containerd/runsc/k8s.io list
```

Having a look with strace about where it blocks:

```
root@ip-172-30-4-239:/var/bin/containerruntimes# strace ./runsc --root /run/containerd/runsc/k8s.io list
[...]
futex(0x3178b20, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x3178b20, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x3178b20, FUTEX_WAIT_PRIVATE, 0, NULL
```

It blocks at acquiring a futex. But not always, sometimes, it blocks when it tries to connect to the problematic sandbox' socket:

```
root@ip-172-30-4-239:/var/bin/containerruntimes# strace ./runsc --root /run/containerd/runsc/k8s.io list
[...]
socket(AF_UNIX, SOCK_STREAM, 0) = 7
connect(7, {sa_family=AF_UNIX, sun_path="/run/containerd/runsc/k8s.io/runsc-ae0b66c097179b61e39a73400d574b4c19812d527d2e2ad9c4dfe92f767246ff.sock"}, 107
```

With this, it is also impossible to run any `runsc debug` command, they would all block either when acquiring the futex or connecting to the sandbox socket.

Any hints on how we can track this down further?

### Steps to reproduce

We are not able to provide exact steps to reproduce the problem. It appears on only one of our many clusters but there on several different nodes. Also, the problem appeared across many different gVisor versions we tried throughout the last year.

### runsc version

```shell
runsc version release-20250820.0
spec: 1.2.0
```

### docker version (if using docker)

```shell
-
```

### uname

Linux ip-172-30-4-239.eu-central-1.compute.internal 6.6.87-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.6.87-0gl0~bp1592 (2025-04-17) x86_64 GNU/Linux

### kubectl (if using Kubernetes)

```shell
$ k version
Client Version: v1.34.1
Kustomize Version: v5.7.1
Server Version: v1.32.7

$ k get node ip-172-30-4-239.eu-central-1.compute.internal
NAME STATUS ROLES AGE VERSION
ip-172-30-4-239.eu-central-1.compute.internal Ready 12d v1.31.7
```

### repo state (if built from source)

_No response_

### runsc debug logs (if available)

```shell

```