OpenImagingLab / OpenImagingLab/FlashVSR

Question about evaluation scripts for quantitative metrics (PSNR / SSIM / LPIPS)

Open
#72 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
152
PR merge metrics
No merged PRs in 30d

Description

Hi, thank you for the great work and for releasing the code and pretrained checkpoints.

I have a question regarding the evaluation of the quantitative metrics reported in the paper, such as PSNR, SSIM, and LPIPS. I was wondering:

Do you have any plan to release the official evaluation / testing scripts used to compute these metrics?

Or alternatively, could you provide some clarification on the exact evaluation protocol?

I have implemented my own evaluation pipeline following common practices. However, when evaluating the official checkpoint, I consistently obtain PSNR values that are about 2 dB lower than those reported in the paper. This makes me suspect that there might be some differences in the evaluation settings.

Any guidance on how these metrics were computed would be greatly appreciated. Thank you very much for your time and help!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by locating the repository's evaluation scripts and checkpoint-testing entry point, then compare their PSNR, SSIM, and LPIPS settings with the paper's protocol; done means the official scripts or exact protocol explains how to reproduce the reported metrics and the approximately 2 dB PSNR difference.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.