"OCI runtime attempted to invoke a command that was not found" error from Podman in sysbox system container

I was trying to follow this post (https://www.redhat.com/sysadmin/podman-inside-kubernetes), and hope to see if I can get Podman working without privileged mode in a K8s pod running Sysbox system container.
I understand there are already existing issues regarding Podman integration into Sysbox so I am here to provide more info:

• https://github.com/nestybox/sysbox/issues/100
• https://github.com/nestybox/sysbox/issues/128

At first I was getting error about /dev/fuse not found and I am aware of the limitation in Sysbox related to this.
I knew I need to have it mounted from host , so I was able to workaround by using the io.kubernetes.cri-o.Devices annotation instead of using the kubelet device plugin as mentioned from the post, and made /dev/fuse with permission 0666. However, I then got the following error when trying to start a container via Podman:

sh-5.2# podman  run --rm hello-world
WARN[0000] Found incomplete layer "942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c", deleting it
Error: crun: make `/var/lib/containers/storage/overlay/edd2911b92f517336aa38932e54809852d2f9c6ab718c7df7a3bfb9bdf587b39/merged` private: No such file or directory: OCI runtime attempted to invoke a command that was not found

But, if I create the pod/container with default runc runtime in CRI-O instead, I was able to get pass that error. Of course, the --privileged flag will not work in this case.

• AWS, Amazon Linux 2, kernel: 5.15.104-63.140.amzn2.x86_64, EKS v1.26
• crio version: 1.26.1
• sysbox version: 0.6.1
• pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: podman
  annotations:
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
    io.kubernetes.cri-o.Devices: /dev/fuse
spec:
  runtimeClassName: sysbox-runc
  containers:
  - name: podman
    image: quay.io/podman/stable
    command: ["sh", "-c", "sleep inf"]
    securityContext:
      capabilities:
        add:
          - "SYS_ADMIN"
          - "MKNOD"
          - "SYS_CHROOT"
          - "SETFCAP"
  restartPolicy: Never

• podman debugged error

sh-5.2# podman --log-level debug run --rm hello-world
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --log-level debug run --rm hello-world)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: imagestore=/var/lib/shared
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=overlayfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 7
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image hello-world (policy: missing)
DEBU[0000] Looking up image "hello-world" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Found image "hello-world" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "hello-world" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76)
WARN[0000] Found incomplete layer "85b6b1ba6b6f2912b269ad09021115d1f378ee3c1f1de3b9c7ebfaba3ff50208", deleting it
DEBU[0000] exporting opaque data as blob "sha256:54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Looking up image "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Found image "quay.io/podman/hello:latest" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "quay.io/podman/hello:latest" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76)
DEBU[0000] exporting opaque data as blob "sha256:54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Looking up image "hello-world" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Found image "hello-world" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "hello-world" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76)
DEBU[0000] exporting opaque data as blob "sha256:54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Inspecting image 54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76
DEBU[0000] exporting opaque data as blob "sha256:54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Inspecting image 54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76
DEBU[0000] Inspecting image 54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76
DEBU[0000] Inspecting image 54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76
DEBU[0000] Inspecting image 54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76
DEBU[0000] using systemd mode: false
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json"
INFO[0000] Sysctl net.ipv4.ping_group_range=0 0 ignored in containers.conf, since Network Namespace set to host
DEBU[0000] Allocated lock 0 for container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] exporting opaque data as blob "sha256:54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Created container "a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807"
DEBU[0000] Container "a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807" has work directory "/var/lib/containers/storage/overlay-containers/a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807/userdata"
DEBU[0000] Container "a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807" has run directory "/run/containers/storage/overlay-containers/a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807/userdata"
DEBU[0000] Not attaching to stdin
INFO[0000] Received shutdown.Stop(), terminating!        PID=242
DEBU[0000] Enabling signal proxying
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/U5CGCP6ZNJKDVUM7RPSUQBVITU,upperdir=/var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/diff,workdir=/var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/work,nodev,fsync=0,volatile
DEBU[0000] Mounted container "a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807" at "/var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged"
DEBU[0000] Created root filesystem for container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 at /var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged
DEBU[0000] Modifying container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 /etc/passwd
DEBU[0000] Modifying container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 /etc/group
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Workdir "/" resolved to host path "/var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged"
DEBU[0000] Created OCI spec for container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 at /var/lib/containers/storage/overlay-containers/a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] Running with no Cgroups
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 -u a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807/userdata -p /run/containers/storage/overlay-containers/a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807/userdata/pidfile -n sad_blackwell --exit-dir /run/libpod/exits --full-attach -l k8s-file:/var/lib/containers/storage/overlay-containers/a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807/userdata/ctr.log --log-level debug --syslog --runtime-arg --cgroup-manager --runtime-arg disabled --conmon-pidfile /run/containers/storage/overlay-containers/a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.imagestore=/var/lib/shared --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,fsync=0 --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1
DEBU[0000] Cleaning up container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Error unmounting /var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged with fusermount3 - exit status 1
DEBU[0000] Failed to remove mountpoint 942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c overlay: /var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged - device or resource busy
DEBU[0000] Unmounted container "a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807"
DEBU[0000] Removing container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807
DEBU[0000] Cleaning up container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 storage is already unmounted, skipping...
DEBU[0000] Removing all exec sessions for container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807
DEBU[0000] Container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 storage is already unmounted, skipping...
DEBU[0000] Failed to delete container "a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807": 1 error occurred:
	* unlinkat /var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c: directory not empty

DEBU[0000] unable to remove container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 after failing to start and attach to it: removing container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 root filesystem: 1 error occurred:
	* unlinkat /var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c: directory not empty

DEBU[0000] ExitCode msg: "crun: make `/var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged` private: no such file or directory: oci runtime attempted to invoke a command that was not found"
Error: crun: make `/var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged` private: No such file or directory: OCI runtime attempted to invoke a command that was not found
DEBU[0000] Shutting down engines

Begin with the supplied pod.yaml and the Podman --log-level debug output; reproduce the failure using the stated EKS, CRI-O, Sysbox, and kernel versions. Compare the crun and runc paths, then consult docs/user-guide/limitations.md. Done means establishing a reproducible compatibility fix or documenting the confirmed limitation.