ResNet101_DUC_HDC can have invalid test_data.
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 9.8k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
During testing models from model zoo I realized that onnxruntime and ngraph give similar results on ResNet101_DUC_HDC but still have 80% missmatch.
Is it possible that test data for ResNet101_DUC_HDC are invalid?
onnxruntime:
E AssertionError:
E Not equal to tolerance rtol=0.001, atol=1e-07
E
E Mismatch: 80.2%
E Max absolute difference: 0.9990995
E Max relative difference: 1.0197404e+09
E x: array([[[3.535608e-04, 1.008540e-03, 6.378655e-04, ..., 5.859119e-02,
E 3.697309e-02, 4.621094e-02],
E [1.310240e-04, 3.463278e-05, 2.679980e-05, ..., 2.111239e-02,...
E y: array([[[8.367564e-12, 4.105623e-11, 2.043494e-11, ..., 1.814237e-10,
E 6.354865e-11, 6.036995e-11],
E [3.100877e-12, 1.409852e-12, 8.585701e-13, ..., 6.537316e-11,...
nGraph:
E AssertionError:
E Not equal to tolerance rtol=0.001, atol=1e-07
E
E Mismatch: 80.2%
E Max absolute difference: 0.99909943
E Max relative difference: 1.0189351e+09
E x: array([[[3.535608e-04, 1.008540e-03, 6.378655e-04, ..., 5.859119e-02,
E 3.697309e-02, 4.621094e-02],
E [1.310240e-04, 3.463278e-05, 2.679980e-05, ..., 2.111239e-02,...
E y: array([[[8.374161e-12, 4.108864e-11, 2.045105e-11, ..., 1.815672e-10,
E 6.359892e-11, 6.041759e-11],
E [3.103328e-12, 1.410971e-12, 8.592453e-13, ..., 6.542469e-11,...
x - reference output
y - actual framework output
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue identifies the ResNet101_DUC_HDC model and reports mismatches from onnxruntime and nGraph against the reference output. Start by reproducing the reported comparison and checking how this model's test data and reference output are produced. Done means establishing whether the test data are invalid and documenting or correcting the affected validation path.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100