onnx / onnx/optimizer

[BUG] The pass "eliminate_nop_monotone_argmax" causes output mismatch

Open
#281 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
834
Forks
109
Avg merge
6h 55m
Merged PRs (30d)
2

Description

[BUG] The pass "eliminate_nop_monotone_argmax" causes output mismatch

Issue
Running the single pass eliminate_nop_monotone_argmax with onnxoptimizer 0.4.2 changes numerical outputs.

Environment

  • Ubuntu 20.04
  • Python 3.10
  • onnx==1.19.0
  • onnxruntime==1.23.2
  • onnxoptimizer==0.4.2(latest)

Repro steps (run from this folder)

eliminate_nop_monotone_argmax_repro.tar.gz

  1. Create a Python environment (Python 3.10) and install dependencies:
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -r requirements.txt
  1. Optimize a case with only eliminate_nop_monotone_argmax (writes model.opt.onnx next to model.onnx):
  • python optimize_model.py --case ./case_03140_seed45116679
  1. Differential test original and optimized models with the same oracle inputs (ORT graph optimization disabled):
  • python diff_test.py --case ./case_03140_seed45116679

Expected
eliminate_nop_monotone_argmax should be semantics-preserving. Applying this pass alone should not change any output values.

Differential Test Output Details

  1. case_03140_seed45116679 — mismatch; output[14] max_abs=4.022e+04 max_rel=4.022e+12, output[15] max_abs=5.190e+02 max_rel=5.120e+10; overall mismatch.
  2. case_00324_seed22816775 — mismatch; output[3] max_abs=5.022e+18 max_rel=1.000e+00; overall mismatch.
  3. case_09334_seed94166965 - mismatch; 1 / 3 of its output nodes are mismatch.
Case: case_03140_seed45116679 (ORT graph optimizations disabled)
  output[0]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 21)
  output[1]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(2, 1, 24)
  output[2]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(42336,)
  output[3]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 21, 24)
  output[4]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(21, 1)
  output[5]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(21, 1)
  output[6]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(21, 1)
  output[7]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(2, 24)
  output[8]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 24)
  output[9]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 24)
  output[10]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 24)
  output[11]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 24)
  output[12]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(21, 24)
  output[13]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(24, 21, 1)
  output[14]: max_abs=4.022e+04, max_rel=4.022e+12, shape=(1, 21, 24)
  output[15]: max_abs=5.190e+02, max_rel=5.120e+10, shape=(32, 21, 24)
Overall: max_abs=4.022e+04, max_rel=4.022e+12


Case: case_00324_seed22816775 (ORT graph optimizations disabled)
  output[0]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 1, 1, 1)
  output[1]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 2, 1, 1)
  output[2]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 1, 1, 51)
  output[3]: max_abs=5.022e+18, max_rel=1.000e+00, shape=(51, 1, 1, 1)
  output[4]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1,)
  output[5]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 2, 1, 1, 51)
  output[6]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(255,)
  output[7]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 2, 2, 51)
  output[8]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 2, 2, 1)
  output[9]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 2, 1, 1, 51)
  output[10]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 2, 1, 1)
  output[11]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 2, 51)
  output[12]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 30, 29, 51)
  output[13]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(51, 29, 30)
Overall: max_abs=5.022e+18, max_rel=1.000e+00


Case: case_09334_seed94166965 (ORT graph optimizations disabled)
  output[0]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 2, 1, 1, 1)
  output[1]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(5, 1, 1, 1, 1)
  output[2]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(2, 1, 62, 1, 1)
  output[3]: max_abs=6.211e-01, max_rel=2.543e-01, shape=(2, 1, 1, 1, 62)
  output[4]: max_abs=1.000e+00, max_rel=1.000e+00, shape=(2, 1, 1, 1, 62)
  output[5]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(58, 1, 1, 1)
  output[6]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(58, 1, 1, 1)
  output[7]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(58, 1, 1, 1)
  output[8]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 58, 1, 62)
  output[9]: max_abs=2.009e-01, max_rel=5.905e-01, shape=(2, 58, 1, 62)
  output[10]: max_abs=3.920e-01, max_rel=3.029e+02, shape=(58, 2, 62, 1)
  output[11]: max_abs=1.792e-01, max_rel=5.003e-01, shape=(2, 58, 1, 62)
  output[12]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(58, 1, 62)
  output[13]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(4, 62)
  output[14]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(1, 58, 1, 62)
  output[15]: max_abs=0.000e+00, max_rel=0.000e+00, shape=(21576,)
Overall: max_abs=1.000e+00, max_rel=3.029e+02

Attachments

  • requirements.txt
  • optimize_model.py
  • diff_test.py (runs original and optimized with same oracle inputs; ORT graph optimizations disabled)
  • run_optimized_model.py (executes optimized model with oracle inputs)
  • case_03140_seed45116679/,case_00324_seed22816775/,case_09334_seed94166965/ (original models and oracle inputs)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running optimize_model.py and diff_test.py from the attached reproduction folder with the listed Python dependencies. Inspect the eliminate_nop_monotone_argmax pass in the optimizer and use the three supplied cases to compare original and optimized models. Done means the pass is semantics-preserving and all reported differential tests show matching outputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
performance, testing-qa, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.