microsoft / microsoft/onnxruntime
[Feature Request] [Compile API] Add support for custom EPContext binary data writer
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
**Problem**
Currently, `ModelCompilationOptions_SetEpContextEmbedMode` allows separating EPContext binary data from the model, but it strictly dumps this data into a separate file. This is problematic for WebNN implementation due to sandbox limitations that prevent direct file system writes.
**Possible Solution**
Custom Writer for EPContext:
Introduces a mechanism similar to `ModelCompilationOptions_SetOutputModelWriteFunc` and `ModelCompilationOptions_SetOutputModelGetInitializerLocationFunc`, allowing users to provide a custom callback function to handle the writing of EPContext binary data.
In-Memory Session Creation:
Adds support for creating an inference session directly from an in-memory model buffer combined with in-memory EPContext binary data.
### Describe scenario use case
This feature is needed for dumping model cache with size >2GB due to the limitation of protobuf.
Contributor guide
Research direction
Start by reading ModelCompilationOptions_SetEpContextEmbedMode, ModelCompilationOptions_SetOutputModelWriteFunc, and ModelCompilationOptions_SetOutputModelGetInitializerLocationFunc in the Compile API. Trace how EPContext data is currently written and how in-memory model buffers are used. Done means a custom EPContext writer and in-memory session creation support the WebNN sandbox use case and model caches larger than 2GB.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend-api-design, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100