Container running with sysbox-runc runtime not responding

Hi,

We're running several containers, where only one of them is run with sysbox-runc runtime. After some time, the container that is running with the sysbox-runc runtime seem to not work properly, and can't be closed (docker rm -f just hangs until timeout is reached).
Meanwhile, the other containers running with the normal runc runtime works properly.

System information:

• Linux ip-172-31-37-92 5.13.0-1017-aws #19~20.04.1-Ubuntu SMP Mon Mar 7 12:53:12 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• Installed shiftfs kernel module based on branch k5.13

Supplying the logs for sysbox-mgr as it does state an error unlike sysbox-fs.
Debug logs from sysbox-mgr.log:

time="2022-04-08 14:00:57" level=info msg="released resources for container 91674898f90d"
time="2022-04-08 14:01:04" level=debug msg="dockerVolMgr: created volume for container 8ec38dddb942"
time="2022-04-08 14:01:04" level=debug msg="k3sVolMgr: created volume for container 8ec38dddb942"
time="2022-04-08 14:01:05" level=debug msg="rke2VolMgr: created volume for container 8ec38dddb942"
time="2022-04-08 14:01:05" level=debug msg="autoRemoveCheck: Docker query start for 8ec38dddb942"
time="2022-04-08 14:01:08" level=debug msg="FIXME: Got an status-code for which error does not match any expected type!!!: -1" module=api status_code=-1
time="2022-04-08 14:01:08" level=debug msg="autoRemoveCheck: Docker query for 8ec38dddb942 failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
time="2022-04-08 14:01:08" level=debug msg="marked shiftfs for /var/lib/docker/overlay2/138b4468e27813eee3e9190b5ed2ca3eece8035c10bac04ea4a1b0bb5b9c5ad8/merged at /var/lib/sysbox/shiftfs/021301a4-d28b-4d0c-a3de-06c12efb7c02"
time="2022-04-08 14:04:22" level=debug msg="unmarked shiftfs for /var/lib/docker/overlay2/138b4468e27813eee3e9190b5ed2ca3eece8035c10bac04ea4a1b0bb5b9c5ad8/merged at /var/lib/sysbox/shiftfs/021301a4-d28b-4d0c-a3de-06c12efb7c02"
time="2022-04-08 14:04:22" level=debug msg="kubeletVolMgr: sync'd-out volume for container 8ec38dddb942"
time="2022-04-08 14:04:22" level=debug msg="k0sVolMgr: sync'd-out volume for container 8ec38dddb942"
time="2022-04-08 14:04:22" level=debug msg="buildkitVolMgr: sync'd-out volume for container 8ec38dddb942"
time="2022-04-08 14:04:22" level=debug msg="containerdVolMgr: sync'd-out volume for container 8ec38dddb942"
time="2022-04-08 14:04:23" level=debug msg="dockerVolMgr: sync'd-out volume for container 8ec38dddb942"
time="2022-04-08 14:04:23" level=debug msg="k3sVolMgr: sync'd-out volume for container 8ec38dddb942"
time="2022-04-08 14:04:23" level=debug msg="rke2VolMgr: sync'd-out volume for container 8ec38dddb942"
time="2022-04-08 14:04:23" level=debug msg="added fs watch on /var/lib/docker/overlay2/138b4468e27813eee3e9190b5ed2ca3eece8035c10bac04ea4a1b0bb5b9c5ad8"
time="2022-04-08 14:04:23" level=info msg="unregistered container 8ec38dddb942"
time="2022-04-08 14:04:24" level=debug msg="rootfsMon: rm on /var/lib/docker/overlay2/138b4468e27813eee3e9190b5ed2ca3eece8035c10bac04ea4a1b0bb5b9c5ad8"
time="2022-04-08 14:04:24" level=debug msg="kubeletVolMgr: destroyed volume for container 8ec38dddb942"
time="2022-04-08 14:04:24" level=debug msg="k0sVolMgr: destroyed volume for container 8ec38dddb942"
time="2022-04-08 14:04:24" level=debug msg="buildkitVolMgr: destroyed volume for container 8ec38dddb942"
time="2022-04-08 14:04:24" level=debug msg="containerdVolMgr: destroyed volume for container 8ec38dddb942"

Core dump of sysbox-mgr:
https://easyupload.io/fg546r

Start by reproducing the container shutdown problem with the sysbox-runc runtime and review sysbox-mgr.log alongside the linked sysbox-mgr core dump. Check the Docker daemon interaction around docker rm -f and the reported shiftfs paths. Done means the affected container remains responsive and can be removed without timing out.