microsoft / microsoft/onnxruntime
[Feature Request] Add more options to load models at InferenceSession constructor
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
The purpose of this feature is to reduce memory usage when loading Onnx models.
Right now InferenceSession constructor is able to load models from two sources:
- string (path to model)
- byte[] (the actual model)
When using the constructors using `Byte[]` , the previous steps typically involve loading the model from a Stream, and in many cases also involve a MemoryStream that has a `.ToArray()` that returns the `byte[]` array of the loaded file.
The problem is that the `.ToArray()` of a `MemoryStream` **creates a copy** of the loaded file because the internal buffers of memory stream are actually larger.
Certainly it could be possible to load the model straight into a `byte[]` array provided you know the file length beforehand, but that's not always possible nor reliable when using certain Streams. So the safe approach to load a stream is to copy it to a MemoryStream and then extract the bytes from it.
MemoryStream has a way to avoid creating a copy, which is using `TryGetBuffer();` that returns an `ArraySegment` , which is what I think the constructors should use instead of `Byte[]`.
So my request is to add additional constructors to InferenceSession:
```c#
public InferenceSession(ArraySegment model);
public InferenceSession(Stream model);
```
### Describe scenario use case
To be able to load models from sources other than file system path or a straight byte[] array, to avoid creating memory copies.
```c#
ArraySegment modelBytes;
using(var m = new MemoryStream())
{
using(var s = await httpClient.GetStreamAsync("model url"))
{
s.CopyTo(m);
}
m.TryGetBuffer(out modelBytes); // get the buffer without creating a copy
}
session = new InferenceSession(modelBytes);
```
Indirectly, this will help lower the memory pressure when using large models on devices with little memory.
Contributor guide
Research direction
Start at the existing InferenceSession constructors that accept a string and byte[] and trace their model-loading path. Compare the requested ArraySegment and Stream entry points with the current byte[] behavior. Done means both new sources can load a model without requiring the extra byte-array copy described in the request.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- api, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100