Mounting host unix sockets to Sysbox containers doesn't always work (e.g. X11 socket forwarding for GUI applications)

Steps to reproduce

In a typical Linux graphical environment, you can open a GUI application to run in a Docker container by forwarding a few files and environment variables from the host to the container, including the X11 unix socket in /tmp/.X11-unix:

$ xhost +local:root
$ docker run -v $XAUTHORITY:/root/.Xauthority -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -e XAUTHORITY=/root/.Xauthority alpine:3.13 sh -c "apk add xeyes && xeyes"
[...]
[GUI app opens]

However, this doesn't work in Sysbox when running in auto userns ID mapping, i.e. the following fails to work:

$ xhost +local:root
$ docker run --runtime=sysbox-runc -v $XAUTHORITY:/root/.Xauthority -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -e XAUTHORITY=/root/.Xauthority alpine:3.13 sh -c "apk add xeyes && xeyes"
[...]
Error: Can't open display: :0

Explanation

The problem happens because the X11 socket (typically /tmp/.X11-unix/X0) is not connectable within the Sysbox container:

$ docker run --init -v /tmp/.X11-unix:/tmp/.X11-unix alpine:3.13 sh -c "apk add netcat-openbsd && nc -U /tmp/.X11-unix/X0"
[No output - OK]

$ docker run --runtime=sysbox-runc --init -v /tmp/.X11-unix:/tmp/.X11-unix alpine:3.13 sh -c "apk add netcat-openbsd && nc -U /tmp/.X11-unix/X0"
nc: unix connect failed: Connection refused

The problem is due to shiftfs not working with UNIX sockets, i.e. UNIX sockets are never connectable on a shiftfs mount. This can be checked because a similar behavior can be reproduced on the host without involving Sysbox, by mounting a shiftfs mark and trying to connect to the socket. This explains why this only happens in auto userns ID mapping mode.

$ mkdir /tmp/.X11-unix-shiftfs
$ sudo mount -t shiftfs -o mark /tmp/.X11-unix /tmp/.X11-unix-shiftfs/
$ nc -U /tmp/.X11-unix/X0
[No output - OK]
$ nc -U /tmp/.X11-unix-shiftfs/X0
nc: unix connect failed: Connection refused
nc: /tmp/.X11-unix-shiftfs/X0: Connection refused

I haven't been able to find any information or functional reason why shiftfs breaks UNIX sockets (e.g. for a reason such as for security), though I'm not well-versed in all those considerations regarding shiftfs. It may just be a limitation.

Workaround

One can work around the issue by chown'ing /tmp/.X11-unix/ to the "sysbox" sub{u,g}id, e.g. as follows assuming it's set to 100000 in /etc/sub{u,g}id:

$ sudo chown 100000:100000 /tmp/.X11-unix/

This works because Sysbox will not mount the directory as a shiftfs but rather do a normal bind-mount if it detects the directory is already owned by the Sysbox {U,G}id. Even though this is a dirty hack, it doesn't appear to have any negative effect neither on the host nor the containers on a typical single-user setup.

Possible fix

I was looking into potential explanations or ways to make the UNIX socket in shiftfs, and while I was looking into this I found that in the past overlayfs mounts also had some trouble connecting to UNIX sockets which have been solved now.

It appears that for the UNIX socket code to work within the Linux kernel, it needs to see the real/underlying inode for the UNIX socket, i.e. a "cloned/mirrored" inode does not work. Nowadays it appears that overlayfs achieves this by having some logic to expose the real/underlying inode instead of a "cloned/mirrored" inode. At first glance I don't think that can even be done for shiftfs since the UID/GID is part of the inode so if we'd expose the real/underlying inode it would not have the shifted UID/GID.

However it appears that it's possible to "recycle" an old overlayfs fix for this by reverting the following commit (there's also some enlightening discussion in the commit message and associated commits):

https://github.com/torvalds/linux/commit/beef5121f3a4d1566c8ab8cd99b4e001862048cf

This is my attempt to reintroduce it for the kernel 5.11.x kernel branch, the permission check is a bit scary though so it'd be useful if this can be properly reviewed:

---
 net/unix/af_unix.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
index 5a31307ceb76..0a94087c0240 100644
--- a/net/unix/af_unix.c
+++ b/net/unix/af_unix.c
@@ -319,7 +319,7 @@ static struct sock *unix_find_socket_byinode(struct inode *i)
 		    &unix_socket_table[i->i_ino & (UNIX_HASH_SIZE - 1)]) {
 		struct dentry *dentry = unix_sk(s)->path.dentry;
 
-		if (dentry && d_backing_inode(dentry) == i) {
+		if (dentry && d_real_inode(dentry) == i) {
 			sock_hold(s);
 			goto found;
 		}
@@ -935,8 +935,8 @@ static struct sock *unix_find_other(struct net *net,
 		err = kern_path(sunname->sun_path, LOOKUP_FOLLOW, &path);
 		if (err)
 			goto fail;
-		inode = d_backing_inode(path.dentry);
-		err = inode_permission(inode, MAY_WRITE);
+		inode = d_real_inode(path.dentry);
+		err = inode_permission(d_backing_inode(path.dentry), MAY_WRITE);
 		if (err)
 			goto put_fail;
 
@@ -1066,7 +1066,7 @@ static int unix_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
 
 	if (sun_path[0]) {
 		addr->hash = UNIX_HASH_SIZE;
-		hash = d_backing_inode(path.dentry)->i_ino & (UNIX_HASH_SIZE - 1);
+		hash = d_real_inode(path.dentry)->i_ino & (UNIX_HASH_SIZE - 1);
 		spin_lock(&unix_table_lock);
 		u->path = path;
 		list = &unix_socket_table[hash];
-- 
2.31.1

Other possible fixes

In case the kernel fix isn't accepted (e.g. allowing UNIX sockets through shiftfs actually introduces some security problem, it's a WONTFIX, etc.) I think this could also be worked around in a cleaner way within Sysbox itself by adding some way to avoid the shiftfs mount for this directory, either in a generic way by providing some way to explicitly ask for a diectory not to be shiftfs-mounted but rather bind-mounted (e.g. by using docker labels), or by just considering a /tmp/.X11-unix mount a special case and not applying shiftfs.