onnx / onnx/optimizer

[BUG] The pass "eliminate_shape_op" causes output mismatch

Open
#262 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
834
Forks
109
Avg merge
6h 55m
Merged PRs (30d)
2

Description

The pass "eliminate_shape_op" causes output mismatch

Issue
eliminate_shape_op (onnxoptimizer 0.4.2) causes output drift in a subset of filtered non-ok cases even when ORT graph optimizations are disabled. case_03181_seed45441358 shows mismatches (v4_0, v25_0), and case_05898_seed66957281 shows NaN mismatches (v14_0).

Environment

  • Ubuntu 20.04
  • Python 3.10
  • onnx==1.19.0
  • onnxruntime==1.19.2
  • onnxoptimizer==0.4.2 (latest)

Repro steps (run from this folder)

  1. Download and unzip the attached archive below, then cd into the extracted directory

eliminate_shape_op_repro.tar.gz

tar -xzvf eliminate_shape_op_repro.tar.gz
cd eliminate_shape_op_repro
  1. Create a Python 3.10 virtual environment and install dependencies:
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
  1. Optimize each case with only eliminate_shape_op:
python optimize_model.py --case case_03181_seed45441358
  1. Run the differential test (ORT optimizations disabled) to compare optimized vs reference:
python diff_test.py --case case_03181_seed45441358

Expected
eliminate_shape_op should be semantics-preserving; applying the pass alone should not change any output values.

Differential Test Output Details

  1. case_03181_seed45441358
Case: case_03181_seed45441358
Mismatched outputs (optimized vs reference):
	v4_0 (output[11]): max_abs=3.352e+00, max_rel=6.005e-01, shape=(1, 135, 17)
	v25_0 (output[12]): max_abs=5.664e+00, max_rel=4.489e+00, shape=(57, 45, 17)
Overall: max_abs=5.664e+00, max_rel=4.489e+00

Attachments

  • README.md (this document)
  • requirements.txt (dependency versions)
  • optimize_model.py (runs only fuse_consecutive_squeezes and saves model.opt.onnx)
  • diff_test.py (runs original vs optimized with oracle inputs and reports max_abs/max_rel)
  • case_03181_seed45441358/ (contains model.onnx original model, oracle.pkl input data for both original and optimized models)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running optimize_model.py for case_03181_seed45441358, then use diff_test.py with ORT graph optimizations disabled to reproduce the mismatch. Read the pass-related behavior described in the repro and compare the optimized model with model.onnx using oracle.pkl. Done means applying eliminate_shape_op alone preserves all output values, including the reported cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.