microsoft / microsoft/onnxruntime
[Feature Request] CPU RAM Offloading for large models
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
Describe the feature request
As per this Huggingface article:
It has a way to load very large models in limited GPU by diving the model into several pieces and swapping them out between the RAM and VRAM.
I'm not sure if this already possible, or being worked on. But this seems like a very sensible idea to incorporate into OnnxRuntime.
Describe scenario use case
This might be useful if say you have a large 10GB model but only 16GB of VRAM.
If you also had 32GB of RAM, then this would make use of that so you could load bigger models.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files, tests, or entry points are identified. Start by reviewing the linked Hugging Face big-modeling guide and determining whether ONNX Runtime already supports moving model pieces between RAM and VRAM; define the design and validation needed to load models larger than available VRAM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100