NVIDIA / NVIDIA/cuda-checkpoint

[vllm dump] need to toggle all the subprocesses before criu dump, it's too slow

Open
#39 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
488
Forks
41
PR merge metrics
No merged PRs in 30d

Description

i just create a vllm service with 2 A10 GPU and use cuda-checkpoint & criu to dump/restore. the model i use is Qwen2.5-72B.

i use the command to start-up:
python3 -m vllm.entrypoints.openai.api_server --model /home/yunzhu.lyq/model14b --host 0.0.0.0 --port 8000 --max-num-seqs 64 --swap-space 4 --gpu-memory-utilization 0.90 --tensor-parallel-size 2 --trust-remote-code --disable-custom-all-reduce >> output.log 2>&1

well, the service contains 5 sub process:

Image

but when i try to dump it, i have to toggle all the processs, which wastes a lot of time, like:
time ./cuda-checkpoint --toggle --pid 1874873
time ./cuda-checkpoint --toggle --pid 1875769
time ./cuda-checkpoint --toggle --pid 1875770
time ./cuda-checkpoint --toggle --pid 1876114
time ./cuda-checkpoint --toggle --pid 1876115

Image

then i can dump them successfully:
time criu dump --shell-job --images-dir images --tree 1874873 --ghost-limit $((10 * 1024 * 1024)) --link-remap --tcp-established

Is this the expected behavior? how can i get faster?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by investigating how cuda-checkpoint --toggle --pid handles a process and whether the command or its process traversal supports descendants. Reproduce the five-process vLLM setup with the commands shown, then verify that one toggle operation covers all subprocesses before the existing criu dump command runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
performance, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.