NVIDIA / NVIDIA/cuda-checkpoint

What is the correct procedure to successfully dump a GPU workload (SGLang) running inside an NVIDIA container with cuda-checkpoint + CRIU?

Open
#52 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
488
Forks
41
PR merge metrics
No merged PRs in 30d

Description

I start an NVIDIA container, then launch an SGLang service inside it.
First I run the CUDA checkpoint pre-freeze command on the target process:

/home/wz/cuda-checkpoint/bin/x86_64_Linux/cuda-checkpoint --toggle --pid 2920058

After that, I execute CRIU dump on the host for the same PID:

sudo /home/wz/criu-host/criu/criu dump --shell-job --images-dir /home/wz/ --tree 2919696

This throws the mount sharing error:

Error (criu/mount.c:1092): mnt: Mount 7894 ./proc/driver/nvidia/gpus/0000🆎00.0 (master_id: 26 shared_id: 0) has unreachable sharing. Try --enable-external-masters.

I added --enable-external-masters --external mnt[] to the CRIU dump command as suggested:

sudo /home/wz/criu-host/criu/criu dump --shell-job --images-dir /home/wz/ --tree 2919696 --enable-external-masters --external mnt[]

Now the dump fails with mapping/plugin errors:

Error (criu/proc_parse.c:118): handle_device_vma plugin failed: No such file or directory
Error (criu/proc_parse.c:672): Can't handle non-regular mapping on 2919696's map 7fc08d672000
Error (criu/cr-dump.c:1589): Collect mappings (pid: 2919696) failed with -1
Error (criu/cr-dump.c:2130): Dumping FAILED.

So what is the correct procedure to successfully dump a GPU workload (SGLang) running inside an NVIDIA container with cuda-checkpoint + CRIU?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the cuda-checkpoint pre-freeze and CRIU dump commands in the issue, then inspect the reported paths in criu/mount.c, criu/proc_parse.c, and criu/cr-dump.c. Compare the behavior with the NVIDIA container and SGLang setup described. Done means documenting a verified command sequence that successfully dumps this GPU workload, or identifying the unsupported mapping or mount requirement.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.