NVIDIA / NVIDIA/cuda-checkpoint
Support checkpointing/restoring GPU memory from file descriptor rather than main memory
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 488
- Forks
- 41
- PR merge metrics
- No merged PRs in 30d
Description
Currently, cuda-checkpoint works by copying GPU memory to main memory (when checkpointing), and copying those pages from main memory back to GPU memory (when restoring). Therefore, cuda-checkpoint requires the application being checkpointed to be able to allocate sufficient main memory to hold a copy of all of its allocated GPU memory. For some applications, this can be quite prohibitively expensive.
Feature request: cuda-checkpoint should support a flag which causes GPU data to be copied to a file on checkpoint, and read from a file on restore. This would allow a caller to control the storage medium where this GPU data is stored.
This should probably be done with O_DIRECT and AIO for performance.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the existing cuda-checkpoint checkpoint and restore paths and command-line flag handling first. Define how file-descriptor storage, O_DIRECT, and AIO should integrate with both operations, then verify that GPU data can be checkpointed to the selected file and restored without requiring equivalent main memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- operating-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100