microsoft / microsoft/onnxruntime
[Feature Request] What does ONNX Runtime do when the model does not fit in the memory?
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
I have a custom accelerator for which I am trying to write an execution provider. I have noticed that the ONNX Runtime transfers all the weights during session initialization. But what happens when the weights do not fit in the memory? Does the ONNX Runtime have the functionality of partitioning the model based on memory? Is there a tool outside of the ONNX Runtime project that partitions the model based on the memory? How do the existing execution providers handle this scenario?
If no such feature of partitioning the model based on memory. Can we have this feature in the future?
### Describe scenario use case
The large language models have parameters that have huge memory requirements, this feature helps deploy these models on edge devices which often have small memory footprint.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reviewing ONNX Runtime's execution-provider and session-initialization behavior, then establish the expected memory-partitioning behavior and how completion would be validated for large models on constrained devices.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100