microsoft / microsoft/onnxruntime
Memory safety for Nvidia GPU time-slicing
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
Nvidia has introduced a feature called time-slicing on GPUs (see [here](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html) and [here](https://docs.nvidia.com/datacenter/cloud-native/openshift/latest/time-slicing-gpus-in-openshift.html)). However, this feature doesn't natively support memory-isolation between replicas- ```Unlike Multi-Instance GPU (MIG), there is no memory or fault-isolation between replicas, but for some workloads this is better than not being able to share at all.```.
As far as I can tell, ONNX doesn't currently have support for safely managing GPU memory while working with GPU time-slicing, as I've seen errors resulting from memory interference. Is safely managing GPU memory while using GPU time-slicing something that folks have considered supporting for ONNX, or have I missed some existing support?
(See https://bruce-lee-ly.medium.com/nvidia-gpu-virtual-memory-management-7fdc4122226b for reference).
### Describe scenario use case
Being able to run multiple small services on a single GPU can lead to cost savings across a broad range of applications, so as to avoid renting more GPUs than necessary. In my particular use case, it would cut the number of required GPUs by about 2/3rds. This also indirectly has some minor environmental benefits, as it reduces the about of required electricity for computation.
Contributor guide
Research direction
The issue does not identify any ONNX Runtime files, tests, or entry points. Begin by locating the GPU memory-management and Nvidia time-slicing support, then determine the expected isolation behavior and how it could be tested before proposing implementation work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, machine-learning
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100