Unable to Start Docker Daemon in Sysbox Container with NFS Mount

Hi, the Docker daemon fails to start within a Sysbox container when the Docker directory is mounted from an NFS server. The issue appears to be related to permissions on the NFS-mounted directory.

On NFS Server

cat /etc/exports

/var/cs/users 10.0.105.0/24(rw,no_subtree_check,all_squash,anonuid=166537,anongid=165536)

Where, 166537 is 165536 + 1001 (1001 is GUI of the user in a Docker container)

ls -lah /var/cs/users/952/u-9524/.ide

total 12K
drwxr-xr-x  5 166537 165536  113 Jun  9 02:05  .
drwxr-xr-x 22 166537 165536 4.0K Jun 13 14:21  ..
drwx--x--- 12 166537 165536  214 Jun 12 18:58  docker
-rw-r--r--  1 166537 165536   35 Jun  9 02:05  info.json

sudo ls -lah /var/cs/users/952/u-9524/.ide/docker

total 4.0K
drwx--x--- 12 166537 165536 214 Jun 12 18:58 .
drwxr-xr-x  5 166537 165536 113 Jun  9 02:05 ..
drwx--x--x  4 166537 165536 170 Jun  9 02:05 buildkit
drwx--x---  2 166537 165536  10 Jun 12 18:58 containers
-rw-------  1 166537 165536  36 Jun  9 02:05 engine-id
drwx------  3 166537 165536  25 Jun  9 02:05 image
drwxr-x---  3 166537 165536  27 Jun  9 02:05 network
drwx------  4 166537 165536  44 Jun  9 02:05 plugins
drwx------  2 166537 165536  10 Jun 12 18:58 runtimes
drwx------  2 166537 165536  10 Jun  9 02:05 swarm
drwx------  2 166537 165536  10 Jun 12 18:58 tmp
drwx--x---  3 166537 165536  25 Jun  9 02:06 vfs
drwx-----x  2 166537 165536  33 Jun  9 02:05 volumes

On Container's Host

cat /etc/docker/daemon.json

{
  "userns-remap": "sysbox",
  "runtimes": {
    "sysbox-runc": {
      "path": "/usr/bin/sysbox-runc"
    }
  },
  "bip": "172.20.0.1/16",
  "default-address-pools": [
    {
      "base": "172.25.0.0/16",
      "size": 24
    }
  ],
  "insecure-registries": [
    "10.0.200.37:5000"
  ]
}

sudo ls -lah /mnt/nfs/users/952/u-9524/.ide

total 12K
drwxr-xr-x  5 166537 165536  113 Jun  9 02:05  .
drwxr-xr-x 22 166537 165536 4.0K Jun 13 14:46  ..
drwx--x--- 12 166537 165536  214 Jun 12 18:58  docker
-rw-r--r--  1 166537 165536   35 Jun  9 02:05  info.json

sudo ls -lah /mnt/nfs/users/952/u-9524/.ide/docker

total 4.0K
drwx--x--- 12 166537 165536 214 Jun 12 18:58 .
drwxr-xr-x  5 166537 165536 113 Jun  9 02:05 ..
drwx--x--x  4 166537 165536 170 Jun  9 02:05 buildkit
drwx--x---  2 166537 165536  10 Jun 12 18:58 containers
-rw-------  1 166537 165536  36 Jun  9 02:05 engine-id
drwx------  3 166537 165536  25 Jun  9 02:05 image
drwxr-x---  3 166537 165536  27 Jun  9 02:05 network
drwx------  4 166537 165536  44 Jun  9 02:05 plugins
drwx------  2 166537 165536  10 Jun 12 18:58 runtimes
drwx------  2 166537 165536  10 Jun  9 02:05 swarm
drwx------  2 166537 165536  10 Jun 12 18:58 tmp
drwx--x---  3 166537 165536  25 Jun  9 02:06 vfs
drwx-----x  2 166537 165536  33 Jun  9 02:05 volumes

Inside a container

ls -lah /home/user/.ide

total 16K
drwxr-xr-x  5 user root  113 Jun  9 02:05  .
drwxr-xr-x 22 user root 4.0K Jun 13 14:41  ..
drwx--x--- 12 user root  214 Jun 12 18:58  docker
-rw-r--r--  1 user root   35 Jun  9 02:05  info.json

sudo ls -lah /home/user/.ide/docker

total 4.0K
drwx--x--- 12 user root 214 Jun 12 18:58 .
drwxr-xr-x  5 user root 113 Jun  9 02:05 ..
drwx--x--x  4 user root 170 Jun  9 02:05 buildkit
drwx--x---  2 user root  10 Jun 12 18:58 containers
-rw-------  1 user root  36 Jun  9 02:05 engine-id
drwx------  3 user root  25 Jun  9 02:05 image
drwxr-x---  3 user root  27 Jun  9 02:05 network
drwx------  4 user root  44 Jun  9 02:05 plugins
drwx------  2 user root  10 Jun 12 18:58 runtimes
drwx------  2 user root  10 Jun  9 02:05 swarm
drwx------  2 user root  10 Jun 12 18:58 tmp
drwx--x---  3 user root  25 Jun  9 02:06 vfs
drwx-----x  2 user root  33 Jun  9 02:05 volumes

sudo systemctl restart docker || journalctl -u docker

Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xeu docker.service" for details.
Jun 13 14:45:05 3c5adbffe1a5 systemd[1]: Starting Docker Application Container Engine...
Jun 13 14:45:05 3c5adbffe1a5 dockerd[104066]: time="2024-06-13T14:45:05.339870626Z" level=info msg="Starting up"
Jun 13 14:45:05 3c5adbffe1a5 dockerd[104066]: could not create or set daemon root permissions: /home/user/.ide/docker: chown /home/user/.ide/docker: operation not permitted
Jun 13 14:45:05 3c5adbffe1a5 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jun 13 14:45:05 3c5adbffe1a5 systemd[1]: docker.service: Failed with result 'exit-code'.
Jun 13 14:45:05 3c5adbffe1a5 systemd[1]: Failed to start Docker Application Container Engine.

Container created with mount

{
  Source: '/mnt/nfs/users/952/u-9524',
  Target: '/home/user',
  Type: 'bind',
  ReadOnly: false,
  BindOptions: {
    Propagation: 'rprivate'
  }
}

Just to give you a little context, we're using NFS share to store Docker data from the container. This way, we can quickly start up our containers and have a shared storage system.

Start with the Docker daemon log showing the chown failure, then compare the host daemon.json, NFS export, and bind-mount configuration provided in the issue. Reproduce the restart with the NFS-backed Docker data directory and determine a supported way for the daemon to start without the permission error.