OpenImagingLab / OpenImagingLab/FlashVSR
Question about evaluation scripts for quantitative metrics (PSNR / SSIM / LPIPS)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 152
- PR merge metrics
- No merged PRs in 30d
Description
Hi, thank you for the great work and for releasing the code and pretrained checkpoints.
I have a question regarding the evaluation of the quantitative metrics reported in the paper, such as PSNR, SSIM, and LPIPS. I was wondering:
Do you have any plan to release the official evaluation / testing scripts used to compute these metrics?
Or alternatively, could you provide some clarification on the exact evaluation protocol?
I have implemented my own evaluation pipeline following common practices. However, when evaluating the official checkpoint, I consistently obtain PSNR values that are about 2 dB lower than those reported in the paper. This makes me suspect that there might be some differences in the evaluation settings.
Any guidance on how these metrics were computed would be greatly appreciated. Thank you very much for your time and help!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating the repository's evaluation scripts and checkpoint-testing entry point, then compare their PSNR, SSIM, and LPIPS settings with the paper's protocol; done means the official scripts or exact protocol explains how to reproduce the reported metrics and the approximately 2 dB PSNR difference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100