microsoft / microsoft/onnxruntime

[Feature Request] What does ONNX Runtime do when the model does not fit in the memory?

Open
#23,664 1 comment 0 reactions 0 assignees View on GitHub
feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

I have a custom accelerator for which I am trying to write an execution provider. I have noticed that the ONNX Runtime transfers all the weights during session initialization. But what happens when the weights do not fit in the memory? Does the ONNX Runtime have the functionality of partitioning the model based on memory? Is there a tool outside of the ONNX Runtime project that partitions the model based on the memory? How do the existing execution providers handle this scenario?

If no such feature of partitioning the model based on memory. Can we have this feature in the future?

### Describe scenario use case

The large language models have parameters that have huge memory requirements, this feature helps deploy these models on edge devices which often have small memory footprint.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing ONNX Runtime's execution-provider and session-initialization behavior, then establish the expected memory-partitioning behavior and how completion would be validated for large models on constrained devices.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.