onnx / onnx/optimizer

[BUG] The pass "eliminate_slice_after_shape" causes great output mismatch

Open
#263 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
834
Forks
109
Avg merge
6h 55m
Merged PRs (30d)
2

Description

The pass "eliminate_slice_after_shape" causes output mismatch

Issue
Running the single pass eliminate_slice_after_shape with onnxoptimizer 0.4.2 may change model outputs. This folder provides minimal repro cases and harnesses to compare original vs optimized models using stored oracle inputs.

Environment

  • Ubuntu 20.04
  • Python 3.10
  • onnx==1.19.0
  • onnxruntime==1.19.2
  • onnxoptimizer==0.4.2 (latest)

Repro steps (run from this folder)

eliminate_slice_after_shape_repro.tar.gz

  1. Create a Python 3.10 virtual environment and install dependencies:
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
  1. Optimize a case with only eliminate_slice_after_shape (writes model.opt.onnx next to model.onnx):
python optimize_model.py --case ./case_02084_seed36754215
  1. Differential test original vs optimized outputs using stored oracle inputs :
python diff_test.py --case ./case_02084_seed36754215
  1. Repeat Step 1 ~ 3 until all the models involved are tested.

Observed results

  • Mismatches:
    • case_03361_seed46866778: output v52_0 drifts (max_abs=6.770e+00, max_rel=1.417e+00).
    • case_05414_seed63124485: output v26_0 drifts (max_abs=1.067e+19, max_rel=2.512e+00).
    • case_05714_seed65500185: output v10_0 drifts (max_abs=1.153e+00, max_rel=1.005e+00).

Expected
eliminate_slice_after_shape should be semantics-preserving. Applying this pass alone should not change any output values.

Differential Test Output Details

  1. case_03361_seed46866778
Case: case_03361_seed46866778
Mismatched outputs (optimized vs reference):
	v52_0 (output[4]): max_abs=6.770e+00, max_rel=1.417e+00, shape=(34, 34, 1, 22)
Overall: max_abs=6.770e+00, max_rel=1.417e+00
  1. case_05414_seed63124485
Case: case_05414_seed63124485
Mismatched outputs (optimized vs reference):
	v26_0 (output[10]): max_abs=1.067e+19, max_rel=2.512e+00, shape=(33, 52)
Overall: max_abs=1.067e+19, max_rel=2.512e+00
  1. case_05714_seed65500185
Case: case_05714_seed65500185
Mismatched outputs (optimized vs reference):
	v10_0 (output[4]): max_abs=1.153e+00, max_rel=1.005e+00, shape=(1, 54)
Overall: max_abs=1.153e+00, max_rel=1.005e+00

Attachments

  • README.md (this document)
  • requirements.txt (dependency versions)
  • optimize_model.py (runs only eliminate_slice_after_shape and saves model.opt.onnx)
  • diff_test.py (runs original vs optimized with oracle inputs and reports max_abs/max_rel)
  • run_optimized_model.py (runs the optimized model to inspect outputs)
  • case_*/ (original models plus oracle inputs for reproduction)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with optimize_model.py and diff_test.py, following the documented virtual-environment setup and testing the listed case_* directories, especially the three cases with mismatches. Use run_optimized_model.py to inspect affected outputs; done means the optimized models preserve the reference outputs for all supplied oracle inputs when only eliminate_slice_after_shape runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning, testing-qa, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.