microsoft / microsoft/onnxruntime

Model saved with offline basic optimizations will not load - ShapeInferenceError

Open
#21,325 1 comment 0 reactions 1 assignee Claimed by @pranavsharma View on GitHub
platform:windows
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

We were attempting to use offline optimization to optimize a model, save it, then load it using later with optimizations disabled.
https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html#onlineoffline-mode

However, the resulting model with basic optimizations results in an error and will not load with ShapeInferenceError

### To reproduce

1. We used Phi Silicia SLM model
2. Save the model offline with basic optimizations - https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html#basic-graph-optimizations
3. In this case we wanted extended data support saved offline so here was the APIs used
```
session_options.SetGraphOptimizationLevel(ORT_ENABLE_BASIC);
const ORTCHAR_T* optimized_model_path = ORT_TSTR("model.extdata.basic_opt.onnx");
session_options.SetOptimizedModelFilePath(optimized_model_path);
session_options.AddConfigEntry(kOrtSessionOptionsOptimizedModelExternalInitializersFileName, "model.extdata.basic_opt.onnx.data");
session_options.AddConfigEntry(kOrtSessionOptionsOptimizedModelExternalInitializersMinSizeInBytes, "10");
```
4. In another code after getting the saved offline model load it
```
session_options.SetGraphOptimizationLevel(ORT_DISABLE_ALL);
session = Ort::Session(env, filemodelpath, session_options);
```
6. The program will exit or at least I didn't see an error
7. If you enable [logging](https://onnxruntime.ai/docs/performance/tune-performance/logging_tracing.html) you will see errors like this "Node (/model/attn_mask_reformat/input_ids_subgraph/Reshape) Op (Reshape) [ShapeInferenceError] Cannot parse data from external tensors. Please load external data into raw data for tensor: /model/attn_mask_reformat/input_ids_subgraph/Concat_3/output_0"

### Urgency

This does block performance investigations, experiments, and workarounds to attempt to work around other ONNX bugs.

### Platform

Windows

### OS Version

11

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.18.1

### ONNX Runtime API

C++

### Architecture

ARM64

### Execution Provider

Default CPU

### Execution Provider Library Version

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.