OpenImagingLab / OpenImagingLab/FlashVSR

About the pixel l2 loss of stage 3

Open
#80 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
152
PR merge metrics
No merged PRs in 30d

Description

Thank you very much for your impressive work. I have been carefully reading the paper and found the design of Stage 3 particularly interesting.

My current understanding is as follows: suppose the two selected latents correspond to temporal indices
[𝑥,𝑥+1].
During decoding, the VAE actually decodes the latent sequence [0,...,x+1 ]to preserve temporal causality, while the latents [0,x-1] are detached from gradients. Then, pixel-domain supervision is applied only to the decoded frames corresponding to the last few frames (e.g., the last 8 frames).

I would like to confirm whether this interpretation is correct, or if instead the decoding is performed only on the selected latents [𝑥,𝑥+1] without explicitly decoding earlier frames.

作者您好,
感谢您的工作为社区带来的贡献,我在尝试应用训练方法到自有数据时,对于阶段3的实现有一点疑问。
在阶段3, 论文中提到具体在pixel domain参与监督的只有随机选择的两个latents,我想知道具体是怎么选择的,
比如选择了时间维度上[x,x+1]的latents的话,做法是否是用VAE decode [0,x+1]的latents,其中[0,x-1]需要detach处理,然后在最后8帧上进行像素域的监督?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file, test, or entry point is named in the issue. Read the Stage 3 training implementation and its VAE decoding path first, then document whether earlier latents are decoded and detached and which selected frames receive pixel-domain supervision.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.