onnx / onnx/models

ResNet101_DUC_HDC can have invalid test_data.

Open
#140 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
9.8k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

During testing models from model zoo I realized that onnxruntime and ngraph give similar results on ResNet101_DUC_HDC but still have 80% missmatch.
Is it possible that test data for ResNet101_DUC_HDC are invalid?

onnxruntime:
E           AssertionError:
E           Not equal to tolerance rtol=0.001, atol=1e-07
E
E           Mismatch: 80.2%
E           Max absolute difference: 0.9990995
E           Max relative difference: 1.0197404e+09
E            x: array([[[3.535608e-04, 1.008540e-03, 6.378655e-04, ..., 5.859119e-02,
E                    3.697309e-02, 4.621094e-02],
E                   [1.310240e-04, 3.463278e-05, 2.679980e-05, ..., 2.111239e-02,...
E            y: array([[[8.367564e-12, 4.105623e-11, 2.043494e-11, ..., 1.814237e-10,
E                    6.354865e-11, 6.036995e-11],
E                   [3.100877e-12, 1.409852e-12, 8.585701e-13, ..., 6.537316e-11,...

nGraph:
E           AssertionError:
E           Not equal to tolerance rtol=0.001, atol=1e-07
E
E           Mismatch: 80.2%
E           Max absolute difference: 0.99909943
E           Max relative difference: 1.0189351e+09
E            x: array([[[3.535608e-04, 1.008540e-03, 6.378655e-04, ..., 5.859119e-02,
E                    3.697309e-02, 4.621094e-02],
E                   [1.310240e-04, 3.463278e-05, 2.679980e-05, ..., 2.111239e-02,...
E            y: array([[[8.374161e-12, 4.108864e-11, 2.045105e-11, ..., 1.815672e-10,
E                    6.359892e-11, 6.041759e-11],
E                   [3.103328e-12, 1.410971e-12, 8.592453e-13, ..., 6.542469e-11,...

x - reference output
y - actual framework output

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue identifies the ResNet101_DUC_HDC model and reports mismatches from onnxruntime and nGraph against the reference output. Start by reproducing the reported comparison and checking how this model's test data and reference output are produced. Done means establishing whether the test data are invalid and documenting or correcting the affected validation path.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.