/proc/<PID>/maps paths are incorrect inside a Sysbox container with shiftfs (breaks apps like dotnet inside the container)

Hi,

I'm using sysbox for build containers to be able to generate Docker images inside a Docker container. Creating container images works, but for some reason the 'dotnet' command used to build C# .NET stuff doesn't.

It fails with the following error message indicating that it is using the wrong path to create a subprocess:

# dotnet publish -warnaserror --configuration Release -o build/server Server

Welcome to .NET 6.0!
---------------------
SDK Version: 6.0.400

Telemetry
---------
The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.

Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
MSBuild version 17.3.0+92e077650 for .NET
  Determining projects to restore...
  Restored /home/jenkins/workspace/<Tool Name>/Server/Server.csproj (in 1.35 sec).
  Restored /home/jenkins/workspace/<Tool Name>/Shared/Shared.csproj (in 2.74 sec).
/usr/share/dotnet/sdk/6.0.400/Roslyn/Microsoft.CSharp.Core.targets(75,5): error MSB6004: The specified task executable location "/var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet" is invalid. [/home/jenkins/workspace/<Tool Name>/Shared/Shared.csproj]

Enabling more verbose output for dotnet shows that it appears to believe its path is /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet instead of just /usr/share/dotnet/dotnet:

MSBuild version 17.3.0+92e077650 for .NET
/usr/share/dotnet/sdk/6.0.400/MSBuild.dll -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/usr/share/dotnet/sdk/6.0.400/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/usr/share/dotnet/sdk/6.0.400/dotnet.dll -maxcpucount -property:PublishDir=/home/jenkins/workspace/<Tool Name>/build/server -property:Configuration=Release -restore -target:Publish -verbosity:m -verbosity:diag -warnaserror Server/Server.csproj
Build started 02/15/2023 08:35:15.
Environment at start of build:
HOSTNAME = be0b24a50b5e
JAVA_HOME = /opt/java/openjdk
LANG = C.UTF-8
DOTNET_HOST_PATH = /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet
MSBuildExtensionsPath = /usr/share/dotnet/sdk/6.0.400/
SHLVL = 1
MSBuildSDKsPath = /usr/share/dotnet/sdk/6.0.400/Sdks
HOME = /root
PATH = /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD = /home/jenkins/workspace/<Tool Name>
MSBuildLoadMicrosoftTargetsReadOnly = true
NODE_VERSION = 16.15.1
OLDPWD = /home/jenkins/workspace
TERM = xterm
_ = /usr/bin/dotnet
JENKINS_AGENT_HOME = /home/jenkins

08:35:15.033     0>Process = "/var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet"
                   MSBuild executable path = "/usr/share/dotnet/sdk/6.0.400/MSBuild.dll"
                   Command line arguments = "/usr/share/dotnet/sdk/6.0.400/MSBuild.dll -maxcpucount -verbosity:m -restore -target:Publish -property:PublishDir=/home/jenkins/workspace/<Tool Name>/build/server -property:Configuration=Release -verbosity:diag -warnaserror Server -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/usr/share/dotnet/sdk/6.0.400/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/usr/share/dotnet/sdk/6.0.400/dotnet.dll"

I've also tried manually setting the environment variable DOTNET_HOST_PATH to the correct value, but it still got overriden to the overlay2 one. When I run the same command using the same build container image without sysbox, the 'dotnet' command works as expected.

The host is a fully-patched Ubuntu 20.04 server using the following kernel: Linux be0b24a50b5e 5.4.0-139-generic #156-Ubuntu SMP Fri Jan 20 17:27:18 UTC 2023 x86_64 GNU/Linux
Sysbox 0.5.2-0.linux is used.

This is the output of the docker info command:

$ sudo docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.16.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 12
  Running: 1
  Paused: 0
  Stopped: 11
 Images: 27
 Server Version: 23.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc sysbox-runc io.containerd.runc.v2
 Default Runtime: sysbox-runc
 Init Binary: docker-init
 containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc version:
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.4.0-139-generic
 Operating System: Ubuntu 20.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.782GiB
 Name: rtb-linuxbuild
 ID: MWHW:2JPI:B4UT:6GQQ:L6Z4:B3IQ:5K66:2ZZ4:VQV5:OO5G:UMYI:QKTZ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.25.0.0/16, Size: 24

WARNING: No swap limit support

Does anyone have any idea what exactly is going on here? It feels like somehow an OverlayFS path is getting exposed to a process in a way it probably shouldn't be.