NVIDIA/cuda-checkpoint

View on GitHub

CUDA checkpoint and restore utility

Stars: 488
Forks: 41
Open beginner issues: 0
Indexed issues: 41

Dominant language: C
License: No license data
Last GitHub push: Jul 6, 2026
Latest indexed: Sep 19, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Beginner friendly Everything

Open All

41 open issues indexed

cuCheckpointProcessLock SIGSEGVs inside libcuda.so.1 on a multi-threaded CUDA service (driver 610.57.04, open kernel module) Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#58 · 3 comments · Opened Sep 16, 2026
Does CUDA IPC support require driver version 610 as a mandatory prerequisite? Open

Difficulty 1/5 Under an hour Newbie friendliness 35/100

NVIDIA/cuda-checkpoint#56 · 1 comment · Opened Jul 27, 2026
Stability concerns when applying cuda-checkpoint --launch-job to existing training workloads Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/cuda-checkpoint#55 · Opened Jul 2, 2026
Killing a cuda-checkpoint restored process hard-hangs the whole host (bare metal, open 595.71.05) — suspending first avoids it Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA/cuda-checkpoint#53 · 1 comment · Opened Jun 11, 2026
What is the correct procedure to successfully dump a GPU workload (SGLang) running inside an NVIDIA container with cuda-checkpoint + CRIU? Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/cuda-checkpoint#52 · 1 comment · Opened Jun 11, 2026
Feature request: GDS (cuFile) or general RDMA transport for checkpoint/restore Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#49 · 1 comment · 1 reaction · Opened May 14, 2026
cuda-checkpoint default restore preserves stale cuDeviceGetPCIBusId cache across cross-node gpu migration; NVML lookups fail as NVML look ups include cudaDeviceGetPciBusId() which returns old id and causes failure post restore Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

NVIDIA/cuda-checkpoint#48 · 5 comments · Opened Apr 22, 2026
Could not restore on process ID 3406143: "invalid argument" Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/cuda-checkpoint#47 · 1 comment · Opened Apr 16, 2026
Is it not possible to restore from GPU 0 to GPU 0? Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/cuda-checkpoint#46 · 3 comments · Opened Apr 9, 2026
cuda-checkpoint --toggle hangs when checkpointing one process of torch.distributed NCCL job Open

Difficulty 4/5 3-5 days Newbie friendliness 30/100

NVIDIA/cuda-checkpoint#45 · 11 reactions · Opened Feb 10, 2026
Restore permanently fails after initial OOM — process stuck in CHECKPOINTED state Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#44 · Opened Jan 28, 2026
Fails with "out of memory" on MIG-enabled GPUs Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA/cuda-checkpoint#42 · 5 comments · 2 reactions · Opened Dec 17, 2025
cuda-checkpoint fails to throw error when performing CRIU on non-CUDA processes Open

Difficulty 4/5 3-5 days Newbie friendliness 30/100

NVIDIA/cuda-checkpoint#41 · 2 comments · Opened Dec 16, 2025
criu failed Open

Difficulty 4/5 3-5 days Newbie friendliness 20/100

NVIDIA/cuda-checkpoint#40 · 5 comments · Opened Oct 24, 2025
[vllm dump] need to toggle all the subprocesses before criu dump, it's too slow Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/cuda-checkpoint#39 · 4 comments · Opened Oct 14, 2025
Request for Dumped Memory Address Exposure in cuCheckpointProcessCheckpoint API Open

Difficulty 5/5 Over a week Newbie friendliness 30/100

NVIDIA/cuda-checkpoint#38 · 2 comments · Opened Sep 24, 2025
[GPU Dependence] "OS call failed or operation not supported on this OS" for `get-state` and `toggle` Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

NVIDIA/cuda-checkpoint#37 · 8 comments · Opened Sep 19, 2025
criu core dump Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#36 · 1 comment · Opened Sep 19, 2025
Cannot get GPU migration to work with CUDA 13 and driver 580 Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#35 · 4 comments · 2 reactions · Opened Aug 28, 2025
"Operation not supported" with GPU driver 570.158.01 on PyTorch Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/cuda-checkpoint#34 · 4 comments · Opened Aug 6, 2025
Support checkpointing/restoring GPU memory from file descriptor rather than main memory Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA/cuda-checkpoint#33 · 6 comments · 14 reactions · Opened Jul 31, 2025
[vLLM] cuda-checkpoint does not Fully Detach File Descriptors to GPU Devices Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/cuda-checkpoint#31 · 2 reactions · Opened Jul 8, 2025
cuda-checkpoint hangs when locking multiprocess nccl-tests Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/cuda-checkpoint#30 · 1 comment · Opened Jun 16, 2025
Slower memory bandwidth than expected. Is this use pinned memory? Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#29 · Opened Jun 10, 2025
`cuda-checkpoint` cannot restore 2-GPU vLLM server process Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA/cuda-checkpoint#27 · 6 comments · 13 reactions · Opened Apr 22, 2025
Simple test counter does not work Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/cuda-checkpoint#25 · 2 comments · Opened Mar 3, 2025
Partial device passthrough workarounds? Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#24 · 3 comments · 1 reaction · Opened Feb 22, 2025
Support for Restoring to a Different GPU After Checkpointing? Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

NVIDIA/cuda-checkpoint#23 · Opened Jan 27, 2025
Graphics workload checkpoint/restore support Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#20 · 3 comments · Opened Nov 14, 2024
Failed to checkpoint dump container using GPU - Unable to connect a transport socket: Permission denied Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#19 · 9 comments · Opened Nov 14, 2024
What does 'does not support GPU migration" mean specifically? Open

Difficulty 1/5 Under an hour Newbie friendliness 20/100

NVIDIA/cuda-checkpoint#18 · 20 comments · 2 reactions · Opened Nov 12, 2024
cuda-checkpoint toggle program does not respond when restoring Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

NVIDIA/cuda-checkpoint#17 · 4 comments · Opened Nov 1, 2024
annot checkpoint container: /usr/bin/nvidia-container-runtime did not terminate successfully: exit status 1 Open

Difficulty 4/5 3-5 days Newbie friendliness 28/100

NVIDIA/cuda-checkpoint#16 · 2 comments · Opened Oct 25, 2024
Support and Limitations of cuda-checkpoint for Distributed Training Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#14 · 3 comments · 1 reaction · Opened Sep 12, 2024
Using cuda-checkpoint with xHPL benchmark Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#13 · 2 comments · Opened Aug 13, 2024
Error on restoring application in docker containers with partial GPU passthrough Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#12 · 4 comments · Opened Aug 1, 2024
"initialization error" on every operation Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#10 · 3 comments · Opened Jun 19, 2024
Will the full code be open source? Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

NVIDIA/cuda-checkpoint#8 · 1 comment · 9 reactions · Opened May 27, 2024
Segmentation fault Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#6 · 4 comments · Opened May 3, 2024
segfault - Error toggling CUDA -- NCCL Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/cuda-checkpoint#5 · 2 reactions · Opened Apr 30, 2024
pytorch support Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/cuda-checkpoint#4 · 25 comments · Opened Apr 30, 2024

NVIDIA/cuda-checkpoint

Get new issues in your inbox