Unable to create containers, "[...] failed to mkdirall [...]" | arm64

Hello,
I get a similar error like https://github.com/nestybox/sysbox/issues/583 . Since the person didn't answer your questions I will start a new issue.

Earlier I got an error for the RaspberryPi 4 model b, https://github.com/nestybox/sysbox/issues/573 suggests to build and install sysbox from source. I did that. I get a different error now that I'm not able to resolve by myself anymore.

Explanation of the error:

When I try to run:
sudo docker run --runtime=sysbox-runc --rm -it --hostname my_cont debian:latest
from your build guide( https://github.com/nestybox/sysbox/blob/4fa4718d9209e93e29a09fd036014673cdf8cc07/docs/developers-guide/build.md ) I get the following error:
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:428: starting container process caused: process_linux.go:607: container init caused: process_linux.go:578: handleReqOp caused: rootfs_init_linux.go:366: failed to mkdirall /var/lib/sysbox/shiftfs/2218d042-3d66-4a5b-8cf8-02faf4973869/sys/devices/virtual/dmi/id: mkdir /var/lib/sysbox/shiftfs/2218d042-3d66-4a5b-8cf8-02faf4973869/sys/devices/virtual/dmi: operation not permitted caused: mkdir /var/lib/sysbox/shiftfs/2218d042-3d66-4a5b-8cf8-02faf4973869/sys/devices/virtual/dmi: operation not permitted: unknown.

I also tried to run a jenkinsAgent container that later should create build- and testing-containers without root permission. Trying to create this container outputs the following error:
Failed to deploy a stack: Network jenkins_inbound_agent_stack_default Creating Network jenkins_inbound_agent_stack_default Created Container jenkinsAgent Creating Container jenkinsAgent Created Container jenkinsAgent Starting Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:428: starting container process caused: process_linux.go:607: container init caused: process_linux.go:578: handleReqOp caused: rootfs_init_linux.go:366: failed to mkdirall /var/lib/sysbox/shiftfs/1d145c4d-8be5-46ac-a6a7-e61adeefd846/sys/devices/virtual/dmi/id: mkdir /var/lib/sysbox/shiftfs/1d145c4d-8be5-46ac-a6a7-e61adeefd846/sys/devices/virtual/dmi: operation not permitted caused: mkdir /var/lib/sysbox/shiftfs/1d145c4d-8be5-46ac-a6a7-e61adeefd846/sys/devices/virtual/dmi: operation not permitted: unknown

Conclusion: Both errors have "failed to mkdirall [...] operation not permitted: unknown.`" in common and therefore I think both errors point to the same root cause.

How to reproduce the error?:

My environment:

• Used OS: Ubuntu server 22.04.1 LTS (Jammy Jellyfish)
• Linux-headers version: 5.15.0-1024-raspi #_26-Ubuntu SMP PREEMPT Wed Jan 18 15:29:53 UTC 2023 aarch64
• Used sysbox release: selfbuild from source 4fa4718d9209e93e29a09fd036014673cdf8cc07
• I have changed the default adress pools in /etc/docker/daemon.json from "172.25.0.0/16" to "171.25.0.0/16"
  after executing sudo ./scr/docker-cfg --sysbox-runtime=enable and getting the following error:

WARN: Docker default-address-pool to configure (172.25.0.0/16) overlaps with existing system subnet. Try another IP range to avoid connectivity issues.
INFO: Refusing to configure Docker networking due to IP address overlap with existing system subnet(s).

The resulting /etc/docker/daemon.json is:

{
    "runtimes": {
        "sysbox-runc": {
            "path": "/usr/bin/sysbox-runc"
        }
    },
    "default-address-pools": [
        {
            "base": "171.25.0.0/16",
            "size": 24
        }
    ]
}

After changing the file and rerunning sudo ./scr/docker-cfg --sysbox-runtime=enable the warning went away.

Steps to reproduce the error:

• Install docker with the installation steps from their website (no snap package!)
• Build and install sysbox on RaspberryPi 4 model b from source with your build guide (link to the build.md file is in the explanation above)
• Change the default adress pool of the /etc/docker/daemon.json file like mentioned above.
• Run sudo docker run --runtime=sysbox-runc --rm -it --hostname my_cont debian:latest from your build guide

My goal and additional information:

The goal is to run a jenkinsAgent container that later should create build- and testing-containers without root permission using sysbox to be able to use containers for testing and building and also be able to run untrusted code.

I don't know whether it is a problem with my system or sysbox and therefore asking for help. The system got initially set up almost 2 years ago. There was a minicube instance installed for testing purposes, is it somehow possible that that interferes with sysbox? Besides this and docker, docker-compose and portainer the system is untouched and standard.

If you need more information please let me know.

Start with the linked developers-guide/build.md instructions, the self-built sysbox revision, and the reported docker run command on the Raspberry Pi environment. Inspect scr/docker-cfg and /etc/docker/daemon.json, then reproduce the mkdirall failure and compare it with the second Jenkins-agent case. Done means identifying the cause and verifying a resolution for the operation-not-permitted error.