NVIDIA / NVIDIA/libnvidia-container
`configure` failure cleanup deletes mount-target files that other instances depend on (shared rootfs)
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 1.1k
- Forks
- 274
- Avg merge
- 1d 4m
- Merged PRs (30d)
- 3
Description
Summary
When nvidia-container-cli configure fails for any reason, its cleanup does not only unmount what it mounted — it also deletes the mount-target files: the fail path in nvc_driver_mount (nvc_mount.c#L800-L804) calls unmount() (nvc_mount.c#L509-L516), which is umount2() followed by file_remove(), and file_remove unlinks every regular file with st_size == 0 (utils.c#L606-L638) — i.e. exactly the placeholder files file_create makes as bind-mount targets.
On a single machine this is safe: unlinking a file that is a mountpoint in another mount namespace fails with EBUSY. But when the container rootfs lives on a shared network filesystem used by several nodes at once (Slurm + enroot-style setups: each node runs configure against the shared rootfs and bind-mounts its node-local driver files over the shared placeholders), the failing node's kernel knows nothing about the other nodes' mounts. The unlinks go through to the server, the other nodes' kernels notice the deletions on the next revalidation and detach their live bind mounts. One failing instance silently breaks GPU userspace (libnvidia-ml.so.1: cannot open shared object file, file too short) on every other node using that rootfs, and nothing on those nodes re-runs configure.
Observed in production (Slurm cluster, shared rootfs on a FUSE-based filesystem, driver 580.159.04, libnvidia-container v1.18.2): two nodes cold-start concurrently; one configure fails; its cleanup unlinks ~55 lib/bin placeholders; the node whose configure succeeded loses all its driver bind mounts seconds later (mountinfo count 9 → 0) and stays broken until pod restart.
Suggested fix
Don't delete mount targets in the failure path — keep the umount2(), drop the file_remove(). The leftover 0-byte placeholders are harmless and actually useful: a retry of configure skips and reuses them (file_create's existing-path handling), and on success they are invisible under the mounts. If leaving placeholders behind after a permanent failure is considered undesirable for private rootfs, an opt-out (e.g. --keep-mount-targets, or a config option) for shared-rootfs deployments would solve it as well.
Note that tracking ownership ("only delete files this run created") would not fix the shared case: concurrent cold-start instances interleave creation, so a failing instance legitimately owns files that other instances have already mounted over.
Happy to send a PR for whichever direction you prefer.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/nvc_mount.c at the failure path in nvc_driver_mount and trace unmount(), then inspect file_remove() and file_create() in src/utils.c. Reproduce or add coverage for a failed configure using shared mount targets, and verify cleanup unmounts mounts without deleting placeholders needed by other instances.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100