NVIDIA / NVIDIA/nvcf

nvsnap: criu-v2 restore depends on the target PID namespace happening to be free

Open
#925 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

nvsnap
Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Problem

criu-v2 restore rebuilds the process tree at the exact PIDs recorded in the
dump, using clone3 with set_tid. It restores into the placeholder pod's
existing PID namespace, so it needs every one of those PIDs to be unused. When
one is not, restore dies at the first collision:

Error (criu/cr-restore.c:1242): Can't fork for 336: File exists

Nothing guarantees they are free. The dump does not serialize a PID namespace,
which the restore log states plainly:

No pidns-1.img image

So correctness here rests on a coincidence: that whatever the placeholder and
the agent have allocated does not overlap the dumped tree's PID range.

Current mitigation

NVIDIA/nvcf#893 pushes the placeholder's next PID allocation to 100000 before
CRIU runs (reservePlaceholderPIDs, internal/agent/restore_v2.go), which the
agent does from the host because a container cannot write
/proc/sys/kernel/ns_last_pid at all. That removes the realistic collision
source and NIM restores reliably with it.

It is still a mitigation. It narrows the window rather than closing it:

  • It is best-effort. If the bump fails the restore proceeds unprotected, by
    design, so that a failure here cannot break restores that would have worked.
  • It assumes no dumped PID exceeds the floor. True today, but it is an
    assumption about workload shape, not an invariant.
  • Anything that allocates in that namespace between the bump and the clone3 can
    still take a PID.

Proposed fix

Dump and restore the PID namespace, so CRIU creates a fresh one and every PID
in it is available by construction. The exact-PID requirement stops being a
constraint on the environment and becomes an internal detail.

This needs design work rather than a flag:

  • The placeholder currently provides the namespace CRIU restores into. With a
    restored PID namespace, what the placeholder is for, and what pid 1 becomes,
    both change.
  • Kubernetes and containerd have their own view of the container's init
    process. A restored PID namespace has to stay consistent with what the
    runtime believes it is supervising, or lifecycle and reaping break.
  • The cgroup placement work (clone3 CLONE_INTO_CGROUP, see restore_v2.go)
    interacts with this and should be revisited alongside it.

Why it matters beyond the current failure

The risk scales with how many processes a workload starts before it settles,
because that determines how high its session leader PID sits. NIM's entrypoint
chain puts it at 336; vLLM and SGLang land far lower. Larger multi-process
serving stacks will sit higher still, so this becomes more likely as workloads
get bigger, not less.

Acceptance criteria

  • A criu-v2 restore succeeds without depending on the target namespace's PID
    allocation state.
  • The ns_last_pid reservation can be removed, or is demonstrably redundant.
  • A regression test restores a tree whose PIDs are deliberately occupied in the
    target namespace beforehand.

References

  • Mitigation and measurements: NVIDIA/nvcf#893
  • Related: NVIDIA/nvcf#892 (repeat capture corrupts the container's overlay)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with internal/agent/restore_v2.go and the reservePlaceholderPIDs mitigation, then trace how CRIU-v2 restore uses the placeholder PID namespace and clone3 cgroup placement. Develop a design that preserves runtime lifecycle behavior and add a regression test restoring with the target PIDs already occupied; done means restoration no longer depends on target namespace allocation state.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, linux
Domain
infrastructure, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.