baidut / baidut/fastiqa

Different results from running the demo

Open
#9 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
4
Forks
0
PR merge metrics
No merged PRs in 30d

Description

I checked out the code with the last commit id ca287e71b7a555191b59a9e9c953be57bbe677cb

I am trying to reproduce the results of the article and of the [demo_paq2piq.ipynb](https://github.com/baidut/fastiqa/blob/master/demo_paq2piq.ipynb) notebook by I get different results.

For example, running the first lines of code of [demo_paq2piq.ipynb](https://github.com/baidut/fastiqa/blob/master/demo_paq2piq.ipynb)
```
from fastiqa.pytorch_standalone_paq2piq import *

model = InferenceModel(RoIPoolModel(), 'https://github.com/baidut/PaQ-2-PiQ/releases/download/v1.0/RoIPoolModel-fit.10.bs.120.pth')
model.blk_size = (3, 5)
model.predict_from_file("images/Picture1.jpg")
```

my scores are 'global_score': 78.89863, 'normalized_global_score': 85.40174865722656 instead of 'global_score': 78.96265,
'normalized_global_score': 85.52611541748047.

I was able to run the scripts using a [NVIDIA docker image of pytorch](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_22-05.html) resulting in the following package versions:
```
NVIDIA Release 22.05 (build 37432893)
PyTorch Version 1.12.0a0+8a1a93a
3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)
[GCC 10.3.0]
fastai.__version__(>= 2.5.3): 2.7.9
fastcore.__version__: 1.5.27
torch.__version__(>= 1.9.1): 1.12.0a0+8a1a93a w/ cuda
torchvision.__version__(>= 0.10.1): 0.13.0a0

{'global_score': 78.89863, 'normalized_global_score': 85.40174865722656, 'local_scores': array([[56.91565 , 52.46511 , 66.20221 , 76.6629 , 58.25135 ],
[77.76071 , 80.47822 , 77.03777 , 77.53446 , 73.78321 ],
[61.88653 , 76.52663 , 77.25575 , 70.014824, 69.26037 ]],
dtype=float32), 'normalized_local_scores': array([[32.2082901 , 20.46800995, 56.70573807, 81.05847168, 35.73180008],
[83.19115448, 88.47036743, 81.78672028, 82.75162506, 75.46418762],
[45.32121277, 80.79373169, 82.21018219, 66.76321411, 64.772995 ]]), 'category': 'Excellent'}
```

Moreover, the code examples for reproducing the results in the paper refer to files not available in the repository
```
LIVE_FB_IQA = load_dbinfo('/home/zq/FB8T/db/LIVE_FB_IQA/dbinfo.json')
CLIVE = load_dbinfo('/home/zq/FB8T/release/CLIVE/dbinfo.json')
KonIQ = load_dbinfo('/home/zq/FB8T/db/KonIQ/dbinfo.json')
```

Am I missing something?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with demo_paq2piq.ipynb at commit ca287e71b7a555191b59a9e9c953be57bbe677cb and rerun the shown InferenceModel/RoIPoolModel example in the reported PyTorch Docker environment. Compare its scores with the notebook output, then inspect the paper reproduction examples and their LIVE_FB_IQA, CLIVE, and KonIQ dbinfo.json paths. Done means the result difference and missing files are explained and the reproducibility instructions are corrected or clarified.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, jupyter-notebook, python, pytorch
Domain
documentation, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.