nvsnap: criu-v2 restore depends on the target PID namespace happening to be free
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Problem
criu-v2 restore rebuilds the process tree at the exact PIDs recorded in the
dump, using clone3 with set_tid. It restores into the placeholder pod's
existing PID namespace, so it needs every one of those PIDs to be unused. When
one is not, restore dies at the first collision:
Error (criu/cr-restore.c:1242): Can't fork for 336: File exists
Nothing guarantees they are free. The dump does not serialize a PID namespace,
which the restore log states plainly:
No pidns-1.img image
So correctness here rests on a coincidence: that whatever the placeholder and
the agent have allocated does not overlap the dumped tree's PID range.
Current mitigation
NVIDIA/nvcf#893 pushes the placeholder's next PID allocation to 100000 before
CRIU runs (reservePlaceholderPIDs, internal/agent/restore_v2.go), which the
agent does from the host because a container cannot write
/proc/sys/kernel/ns_last_pid at all. That removes the realistic collision
source and NIM restores reliably with it.
It is still a mitigation. It narrows the window rather than closing it:
- It is best-effort. If the bump fails the restore proceeds unprotected, by
design, so that a failure here cannot break restores that would have worked. - It assumes no dumped PID exceeds the floor. True today, but it is an
assumption about workload shape, not an invariant. - Anything that allocates in that namespace between the bump and the clone3 can
still take a PID.
Proposed fix
Dump and restore the PID namespace, so CRIU creates a fresh one and every PID
in it is available by construction. The exact-PID requirement stops being a
constraint on the environment and becomes an internal detail.
This needs design work rather than a flag:
- The placeholder currently provides the namespace CRIU restores into. With a
restored PID namespace, what the placeholder is for, and what pid 1 becomes,
both change. - Kubernetes and containerd have their own view of the container's init
process. A restored PID namespace has to stay consistent with what the
runtime believes it is supervising, or lifecycle and reaping break. - The cgroup placement work (clone3 CLONE_INTO_CGROUP, see restore_v2.go)
interacts with this and should be revisited alongside it.
Why it matters beyond the current failure
The risk scales with how many processes a workload starts before it settles,
because that determines how high its session leader PID sits. NIM's entrypoint
chain puts it at 336; vLLM and SGLang land far lower. Larger multi-process
serving stacks will sit higher still, so this becomes more likely as workloads
get bigger, not less.
Acceptance criteria
- A criu-v2 restore succeeds without depending on the target namespace's PID
allocation state. - The
ns_last_pidreservation can be removed, or is demonstrably redundant. - A regression test restores a tree whose PIDs are deliberately occupied in the
target namespace beforehand.
References
- Mitigation and measurements: NVIDIA/nvcf#893
- Related: NVIDIA/nvcf#892 (repeat capture corrupts the container's overlay)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with internal/agent/restore_v2.go and the reservePlaceholderPIDs mitigation, then trace how CRIU-v2 restore uses the placeholder PID namespace and clone3 cgroup placement. Develop a design that preserves runtime lifecycle behavior and add a regression test restoring with the target PIDs already occupied; done means restoration no longer depends on target namespace allocation state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, linux
- Domain
- infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100