OpenImagingLab / OpenImagingLab/FlashVSR

missalignment problem

Open
#46 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
152
PR merge metrics
No merged PRs in 30d

Description

Thanks for opening source such great work! I have a problem about the missalignment between the generated video and the input low quality video. According to the largest_8n1_leq function the length of the low quallity video is 8n+1, while the length of the noise input to the diffusion model is 2n, corresponding to 8n-3 frames of the generated video. Moreover, the first 6 noise takes first 25 frames of low quality video as condition and each subsequent noise corresponds to 4 frames of low-resolution video. 8n+1 frames of low quality video are needed as input to the diffusion model. While when the latents are converted into video by the TCDecoder, only 8n-3 frames of low quality video are needed. Why is there a missalignment between the input video and the generated video?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the frame indexing from largest_8n1_leq in examples/WanVSR/infer_flashvsr_v1.1_tiny.py, then compare it with the TCDecoder call in diffsynth/pipelines/flashvsr_tiny.py. Reproduce the stated 8n+1 versus 8n-3 inputs and determine whether the difference is intentional or an alignment bug. Done means the frame mapping is explained and any required correction is covered by a reproducible check.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.