OpenImagingLab / OpenImagingLab/FlashVSR
About the pixel l2 loss of stage 3
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 152
- PR merge metrics
- No merged PRs in 30d
Description
Thank you very much for your impressive work. I have been carefully reading the paper and found the design of Stage 3 particularly interesting.
My current understanding is as follows: suppose the two selected latents correspond to temporal indices
[𝑥,𝑥+1].
During decoding, the VAE actually decodes the latent sequence [0,...,x+1 ]to preserve temporal causality, while the latents [0,x-1] are detached from gradients. Then, pixel-domain supervision is applied only to the decoded frames corresponding to the last few frames (e.g., the last 8 frames).
I would like to confirm whether this interpretation is correct, or if instead the decoding is performed only on the selected latents [𝑥,𝑥+1] without explicitly decoding earlier frames.
作者您好,
感谢您的工作为社区带来的贡献,我在尝试应用训练方法到自有数据时,对于阶段3的实现有一点疑问。
在阶段3, 论文中提到具体在pixel domain参与监督的只有随机选择的两个latents,我想知道具体是怎么选择的,
比如选择了时间维度上[x,x+1]的latents的话,做法是否是用VAE decode [0,x+1]的latents,其中[0,x-1]需要detach处理,然后在最后8帧上进行像素域的监督?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file, test, or entry point is named in the issue. Read the Stage 3 training implementation and its VAE decoding path first, then document whether earlier latents are decoded and detached and which selected frames receive pixel-domain supervision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100