OpenImagingLab / OpenImagingLab/FlashVSR

How do you align output and GT for reference metrics when sizes differ?

Open
#109 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
152
PR merge metrics
No merged PRs in 30d

Description

Thanks for the great work and for releasing the code.

I have a question about the evaluation protocol for the full-reference metrics, specifically how output/GT resolution mismatch is handled.

The inference pipeline requires the network input to be a multiple of 128 in each spatial dimension. As far as I can tell,
examples/WanVSR/infer_flashvsr_v1.1_tiny.py
tW = (sW // multiple) * multiple # multiple = 128
tH = (sH // multiple) * multiple
upscale_then_center_crop(...)

Because of this floor-crop, the SR output is not the same size as the original ground truth. For example, on REDS (GT 1280×720):

LR input: 320×180
×4 upscale: 1280×720
floor to multiple of 128: 1280×640 (width 1280 is already 10×128, but height 720 → 640)
So the output loses 80 px in height relative to the GT.

Question
The paper reports PSNR/SSIM/LPIPS on YouHQ40, REDS, and SPMCS, but I couldn't find how the output and GT are spatially aligned before computing these pixel-aligned metrics. When the output size (e.g. 1280×640) differs from the GT size (e.g. 1280×720):

How do you align them for the reference metrics — do you center-crop the GT to the output size, pad the output back, or something else?
Are the reported PSNR/SSIM/LPIPS computed on the cropped region only, or on the full GT frame?
If baselines produce full-resolution outputs while FlashVSR is evaluated on a cropped region, how is a fair comparison ensured?
Could you share the exact evaluation alignment step (ideally the script) so the numbers can be reproduced and compared fairly? Thanks!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with examples/WanVSR/infer_flashvsr_v1.1_tiny.py and trace how the output dimensions are produced after the multiple-of-128 crop. Search the repository for the PSNR, SSIM, and LPIPS evaluation entry points; done means the spatial alignment step and evaluation region are documented well enough to reproduce and compare the reported metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.