NVIDIA / NVIDIA/cuda-checkpoint
[vllm dump] need to toggle all the subprocesses before criu dump, it's too slow
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 488
- Forks
- 41
- PR merge metrics
- No merged PRs in 30d
Description
i just create a vllm service with 2 A10 GPU and use cuda-checkpoint & criu to dump/restore. the model i use is Qwen2.5-72B.
i use the command to start-up:
python3 -m vllm.entrypoints.openai.api_server --model /home/yunzhu.lyq/model14b --host 0.0.0.0 --port 8000 --max-num-seqs 64 --swap-space 4 --gpu-memory-utilization 0.90 --tensor-parallel-size 2 --trust-remote-code --disable-custom-all-reduce >> output.log 2>&1
well, the service contains 5 sub process:
but when i try to dump it, i have to toggle all the processs, which wastes a lot of time, like:
time ./cuda-checkpoint --toggle --pid 1874873
time ./cuda-checkpoint --toggle --pid 1875769
time ./cuda-checkpoint --toggle --pid 1875770
time ./cuda-checkpoint --toggle --pid 1876114
time ./cuda-checkpoint --toggle --pid 1876115
then i can dump them successfully:
time criu dump --shell-job --images-dir images --tree 1874873 --ghost-limit $((10 * 1024 * 1024)) --link-remap --tcp-established
Is this the expected behavior? how can i get faster?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by investigating how cuda-checkpoint --toggle --pid handles a process and whether the command or its process traversal supports descendants. Reproduce the five-process vLLM setup with the commands shown, then verify that one toggle operation covers all subprocesses before the existing criu dump command runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- performance, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100