Cannot mount VM shared folders inside docker system container

I guess I'm not 100% sure this is actually a supported use case, but I couldn't find evidence either way.

My usage for sysbox is to be able to run a VM (using vagrant) which can then use sysbox to run system containers, that can then be used to test my application running inside the system container
(that needs to use docker and systemd directly). Right now running on an Ubuntu machine, but the thinking with the VM is to be able to support using a MacOS host as well.

Everything seems to work, except for setting correct IDs for bind mounts (from VM filesystem to inner docker filesystem). Looking at the logs for sysbox-mgr, it seems like none of the methods for ID-mapping for mounts succeed:

Jul 11 20:09:09 ubuntu2204.localdomain systemd[1]: Starting sysbox-mgr (part of the Sysbox container runtime)...
Jul 11 20:09:09 ubuntu2204.localdomain sysbox-mgr[9948]: time="2023-07-11 20:09:09" level=info msg="Starting ..."
Jul 11 20:09:09 ubuntu2204.localdomain sysbox-mgr[9948]: time="2023-07-11 20:09:09" level=info msg="Sysbox data root: /var/lib/sysbox"
Jul 11 20:09:09 ubuntu2204.localdomain sysbox-mgr[9948]: time="2023-07-11 20:09:09" level=info msg="Shiftfs module found in kernel: yes"
Jul 11 20:09:09 ubuntu2204.localdomain sysbox-mgr[9948]: time="2023-07-11 20:09:09" level=info msg="Shiftfs works properly: no"
Jul 11 20:09:09 ubuntu2204.localdomain sysbox-mgr[9948]: time="2023-07-11 20:09:09" level=info msg="Shiftfs-on-overlayfs works properly: no"
Jul 11 20:09:09 ubuntu2204.localdomain sysbox-mgr[9948]: time="2023-07-11 20:09:09" level=info msg="ID-mapped mounts supported by kernel: yes"
Jul 11 20:09:09 ubuntu2204.localdomain sysbox-mgr[9948]: time="2023-07-11 20:09:09" level=info msg="Overlayfs on ID-mapped mounts supported by kernel: no"
Jul 11 20:09:09 ubuntu2204.localdomain sysbox-mgr[9948]: time="2023-07-11 20:09:09" level=info msg="Operating in system container mode."
Jul 11 20:09:09 ubuntu2204.localdomain sysbox-mgr[9948]: time="2023-07-11 20:09:09" level=info msg="Inner container image preloading enabled."
Jul 11 20:09:09 ubuntu2204.localdomain sysbox-mgr[9948]: time="2023-07-11 20:09:09" level=info msg="Listening on /run/sysbox/sysmgr.sock"
Jul 11 20:09:09 ubuntu2204.localdomain systemd[1]: Started sysbox-mgr (part of the Sysbox container runtime).
Jul 11 20:09:09 ubuntu2204.localdomain sysbox-mgr[9948]: time="2023-07-11 20:09:09" level=info msg="Ready ..."

Tested using both virtualbox and libvirt (kvm) virtualization for vagrant.

Host Machine

Ubuntu 22.04 with 5.17 kernel

VM Machine

Ubuntu 22.04 with 5.15 kernel + ShiftFS installed
Sysbox version 0.6.1
Docker version 24.0.4

Recreate

# Using virtualbox
$ vagrant up
# Using libvirt
$ vagrant up --provider=libvirt
$ vagrant ssh
# Inside VM
$ docker run --runtime=sysbox-runc -it -v /home/vagrant:/root/ws ubuntu:focal /bin/bash
# Inside Container
$ ls -la /root/
# Should see
total 24
drwx------ 1 root   root    4096 Jul 11 20:41 .
drwxr-xr-x 1 root   root    4096 Jul 11 20:41 ..
-rw-r--r-- 1 root   root    3106 Dec  5  2019 .bashrc
-rw-r--r-- 1 root   root     161 Dec  5  2019 .profile
drwxr-x--- 6 nobody nogroup 4096 Jul 11 20:11 ws

This will happen for mounting folders from the VM disk into the docker container, as well as for synced VM folders (through either virtualbox or libvirtio-fs).

$script = <<-SCRIPT
set -euxo pipefail
export DEBIAN_FRONTEND=noninteractive

echo "Installing docker"
apt-get update
apt-get install -y ca-certificates curl gnupg
mkdir -p /etc/apt/keyrings/
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --batch --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" > \
  /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
usermod -a -G docker vagrant

# Install shiftfs
apt-get install -y make dkms git wget
git clone -b k5.16 https://github.com/toby63/shiftfs-dkms.git shiftfs-k516
pushd shiftfs-k516
./update1
make -f Makefile.dkms
modinfo shiftfs
popd

echo "Installing sysbox"
# From https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md
wget -q https://downloads.nestybox.com/sysbox/releases/v0.6.1/sysbox-ce_0.6.1-0.linux_amd64.deb
apt-get install -y jq
apt-get install -y ./sysbox-ce_0.6.1-0.linux_amd64.deb
rm sysbox-ce_0.6.1-0.linux_amd64.deb

echo "Done"
SCRIPT


Vagrant.configure("2") do |config|
  config.vm.box = "generic/ubuntu2204"

  config.vm.provision "shell", inline: $script

  config.vm.synced_folder "./", "/home/vagrant/ws/", type: "virtiofs"

  config.ssh.forward_agent = true
  
  config.vm.provider :libvirt do |libvirt|
    libvirt.cpus = 4
    libvirt.memory = 8192
    libvirt.memorybacking :access, :mode => "shared"
  end
end

$script = <<-SCRIPT
set -euxo pipefail
export DEBIAN_FRONTEND=noninteractive

echo "Installing docker"
apt-get update
apt-get install -y ca-certificates curl gnupg
mkdir -p /etc/apt/keyrings/
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --batch --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" > \
  /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
usermod -a -G docker vagrant

# Install shiftfs
apt-get install -y make dkms git wget
git clone -b k5.16 https://github.com/toby63/shiftfs-dkms.git shiftfs-k516
pushd shiftfs-k516
./update1
make -f Makefile.dkms
modinfo shiftfs
popd

echo "Installing sysbox"
# From https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md
wget -q https://downloads.nestybox.com/sysbox/releases/v0.6.1/sysbox-ce_0.6.1-0.linux_amd64.deb
apt-get install -y jq
apt-get install -y ./sysbox-ce_0.6.1-0.linux_amd64.deb
rm sysbox-ce_0.6.1-0.linux_amd64.deb

echo "Done"
SCRIPT


Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/jammy64"

  config.vm.provision "shell", inline: $script

  config.vm.synced_folder "./", "/home/vagrant/ws/"

  config.ssh.forward_agent = true

  config.vm.provider "virtualbox" do |v|
    v.memory = 4096
    v.cpus = 8
  end
end

Start with the documented reproduction: provision the Ubuntu 22.04 VM, install Sysbox 0.6.1, and run the shown docker command with /home/vagrant mounted. Compare the sysbox-mgr capability logs with the resulting nobody:nogroup ownership, testing both regular and Vagrant-synced folders. Done means determining whether these mounts are supported and either restoring the expected IDs or documenting the limitation.