Different results from running the demo
- Dominant language
- Jupyter Notebook
- Stars
- 4
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
I checked out the code with the last commit id ca287e71b7a555191b59a9e9c953be57bbe677cb
I am trying to reproduce the results of the article and of the [demo_paq2piq.ipynb](https://github.com/baidut/fastiqa/blob/master/demo_paq2piq.ipynb) notebook by I get different results.
For example, running the first lines of code of [demo_paq2piq.ipynb](https://github.com/baidut/fastiqa/blob/master/demo_paq2piq.ipynb)
```
from fastiqa.pytorch_standalone_paq2piq import *
model = InferenceModel(RoIPoolModel(), 'https://github.com/baidut/PaQ-2-PiQ/releases/download/v1.0/RoIPoolModel-fit.10.bs.120.pth')
model.blk_size = (3, 5)
model.predict_from_file("images/Picture1.jpg")
```
my scores are 'global_score': 78.89863, 'normalized_global_score': 85.40174865722656 instead of 'global_score': 78.96265,
'normalized_global_score': 85.52611541748047.
I was able to run the scripts using a [NVIDIA docker image of pytorch](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_22-05.html) resulting in the following package versions:
```
NVIDIA Release 22.05 (build 37432893)
PyTorch Version 1.12.0a0+8a1a93a
3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)
[GCC 10.3.0]
fastai.__version__(>= 2.5.3): 2.7.9
fastcore.__version__: 1.5.27
torch.__version__(>= 1.9.1): 1.12.0a0+8a1a93a w/ cuda
torchvision.__version__(>= 0.10.1): 0.13.0a0
{'global_score': 78.89863, 'normalized_global_score': 85.40174865722656, 'local_scores': array([[56.91565 , 52.46511 , 66.20221 , 76.6629 , 58.25135 ],
[77.76071 , 80.47822 , 77.03777 , 77.53446 , 73.78321 ],
[61.88653 , 76.52663 , 77.25575 , 70.014824, 69.26037 ]],
dtype=float32), 'normalized_local_scores': array([[32.2082901 , 20.46800995, 56.70573807, 81.05847168, 35.73180008],
[83.19115448, 88.47036743, 81.78672028, 82.75162506, 75.46418762],
[45.32121277, 80.79373169, 82.21018219, 66.76321411, 64.772995 ]]), 'category': 'Excellent'}
```
Moreover, the code examples for reproducing the results in the paper refer to files not available in the repository
```
LIVE_FB_IQA = load_dbinfo('/home/zq/FB8T/db/LIVE_FB_IQA/dbinfo.json')
CLIVE = load_dbinfo('/home/zq/FB8T/release/CLIVE/dbinfo.json')
KonIQ = load_dbinfo('/home/zq/FB8T/db/KonIQ/dbinfo.json')
```
Am I missing something?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with demo_paq2piq.ipynb at commit ca287e71b7a555191b59a9e9c953be57bbe677cb and rerun the shown InferenceModel/RoIPoolModel example in the reported PyTorch Docker environment. Compare its scores with the notebook output, then inspect the paper reproduction examples and their LIVE_FB_IQA, CLIVE, and KonIQ dbinfo.json paths. Done means the result difference and missing files are explained and the reproducibility instructions are corrected or clarified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, jupyter-notebook, python, pytorch
- Domain
- documentation, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100