/run/nvidia-persistenced/socket: no such device or address

Start with the enroot log and the /raid/enroot//hooks.d/98-nvidia.sh entry point, then reproduce the GPU image launch with the shown srun command. Inspect how nvidia-container-cli handles /run/nvidia-persistenced/socket and compare it with the successful CPU image launch. Done means the GPU-based container starts without the socket mount error.

Hi,
I was referenced here from
https://github.com/NVIDIA/libnvidia-container/issues/187

We are currently setting up a new cluster deployment environment with slurm, pyxis and enroot.
Our machines have DGX OS installed.

Container images like centos
srun --container-image=centos grep PRETTY /etc/os-release

finish without a problem. GPU Based images like
srun --gpu=1 --container-image=nvcr.io/nvidia/tensorflow:22.08-tf2-py3 /bin/bash

experiencing a problem during the startup:

slurmstepd: error: pyxis: container start failed with error code: 1
slurmstepd: error: pyxis: printing enroot log file:
slurmstepd: error: pyxis:     nvidia-container-cli: mount error: file creation failed: /raid/enroot-data/user-9011/pyxis_64.0/run/nvidia-persistenced/socket: no such device or address
slurmstepd: error: pyxis:     [ERROR] /raid/enroot//hooks.d/98-nvidia.sh exited with return code 1
slurmstepd: error: pyxis: couldn't start container

I think it is some misconfiguration, but at the moment I am not able to spot it.

enroot verify:

Linux version 5.4.0-117-generic (buildd@lcy02-amd64-006) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) #132-Ubuntu SMP Thu Jun 2 00:39:06 UTC 2022

Kernel configuration:

CONFIG_NAMESPACES                 : OK
CONFIG_USER_NS                    : OK
CONFIG_SECCOMP_FILTER             : OK
CONFIG_OVERLAY_FS                 : OK (module)
CONFIG_X86_VSYSCALL_EMULATION     : OK
CONFIG_VSYSCALL_EMULATE           : KO (required if glibc <= 2.13)
CONFIG_VSYSCALL_NATIVE            : KO (required if glibc <= 2.13)

Kernel command line:

vsyscall=native                   : KO (required if glibc <= 2.13)
vsyscall=emulate                  : KO (required if glibc <= 2.13)

Kernel parameters:

kernel.unprivileged_userns_clone  : OK
user.max_user_namespaces          : OK
user.max_mnt_namespaces           : OK

Extra packages:

nvidia-container-cli              : OK

GLIBC Version:

ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9.9) 2.31

/lib/x86_64-linux-gnu/libc.so.6 --version
GNU C Library (Ubuntu GLIBC 2.31-0ubuntu9.9) stable release version 2.31.
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 9.4.0.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://bugs.launchpad.net/ubuntu/+source/glibc/+bugs>.