microsoft / microsoft/onnxruntime

[Feature Request] Add more options to load models at InferenceSession constructor

Open
#23,940 0 comments 0 reactions 0 assignees View on GitHub
api:CSharp feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

The purpose of this feature is to reduce memory usage when loading Onnx models.

Right now InferenceSession constructor is able to load models from two sources:

- string (path to model)
- byte[] (the actual model)

When using the constructors using `Byte[]` , the previous steps typically involve loading the model from a Stream, and in many cases also involve a MemoryStream that has a `.ToArray()` that returns the `byte[]` array of the loaded file.

The problem is that the `.ToArray()` of a `MemoryStream` **creates a copy** of the loaded file because the internal buffers of memory stream are actually larger.

Certainly it could be possible to load the model straight into a `byte[]` array provided you know the file length beforehand, but that's not always possible nor reliable when using certain Streams. So the safe approach to load a stream is to copy it to a MemoryStream and then extract the bytes from it.

MemoryStream has a way to avoid creating a copy, which is using `TryGetBuffer();` that returns an `ArraySegment` , which is what I think the constructors should use instead of `Byte[]`.

So my request is to add additional constructors to InferenceSession:

```c#
public InferenceSession(ArraySegment model);
public InferenceSession(Stream model);
```

### Describe scenario use case

To be able to load models from sources other than file system path or a straight byte[] array, to avoid creating memory copies.

```c#
ArraySegment modelBytes;

using(var m = new MemoryStream())
{
using(var s = await httpClient.GetStreamAsync("model url"))
{
s.CopyTo(m);
}

m.TryGetBuffer(out modelBytes); // get the buffer without creating a copy
}

session = new InferenceSession(modelBytes);
```

Indirectly, this will help lower the memory pressure when using large models on devices with little memory.

Contributor guide

Open the contributing guide

Research direction

Start at the existing InferenceSession constructors that accept a string and byte[] and trace their model-loading path. Compare the requested ArraySegment and Stream entry points with the current byte[] behavior. Done means both new sources can load a model without requiring the extra byte-array copy described in the request.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
api, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.