microsoft / microsoft/onnxruntime

[Feature Request] [Compile API] Add support for custom EPContext binary data writer

Open
#27,310 0 comments 0 reactions 0 assignees View on GitHub
ep:WebNN feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

**Problem**

Currently, `ModelCompilationOptions_SetEpContextEmbedMode` allows separating EPContext binary data from the model, but it strictly dumps this data into a separate file. This is problematic for WebNN implementation due to sandbox limitations that prevent direct file system writes.

**Possible Solution**

Custom Writer for EPContext:
Introduces a mechanism similar to `ModelCompilationOptions_SetOutputModelWriteFunc` and `ModelCompilationOptions_SetOutputModelGetInitializerLocationFunc`, allowing users to provide a custom callback function to handle the writing of EPContext binary data.

In-Memory Session Creation:
Adds support for creating an inference session directly from an in-memory model buffer combined with in-memory EPContext binary data.

### Describe scenario use case

This feature is needed for dumping model cache with size >2GB due to the limitation of protobuf.

Contributor guide

Open the contributing guide

Research direction

Start by reading ModelCompilationOptions_SetEpContextEmbedMode, ModelCompilationOptions_SetOutputModelWriteFunc, and ModelCompilationOptions_SetOutputModelGetInitializerLocationFunc in the Compile API. Trace how EPContext data is currently written and how in-memory model buffers are used. Done means a custom EPContext writer and in-memory session creation support the WebNN sandbox use case and model caches larger than 2GB.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend-api-design, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.