RIFE4.25微调和一些代码疑问
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
作者您好,我在预训练的Rife v4.25模型基础上,用 vimeo-90k 随机采样 25% 约 13000 个样本作为训练数据,学习率[1e-7,1e-5],微调了3~5 个 epoch。
然后用inference_video插帧视频时边缘会有水波状扭曲,以及运动物体重影,看起来模型的细节处理和运动光流估计能力都被破坏了。
train过程中的loss一直震荡不收敛,validate的lpips逐渐下降
是训练轮数不够还是数据集不足,或者是训练策略的问题?
以及关于RIFE4.25训练代码的一些问题:
-
dataset.py中64到67行
if img0.shape[0] < 448:
img0 = np.concatenate((img0, img0[:, :, ::-1].copy()), 0)
img1 = np.concatenate((img1, img1[:, :, ::-1].copy()), 0)
gt = np.concatenate((gt, gt[:, :, ::-1].copy()), 0)
这里为什么要交换颜色通道?似乎是笔误想翻转高度?而且这样拼接导致画面不连续了 -
model.py中Model的update函数返回值
return merged[-1], {
'merged_tea': teacher_res[0][0],
'mask': mask,
'mask_tea': mask,
'flow': flow[3][:, :2],
为什么返回的是倒数第二层的flow?最后一层的不是更加直观吗
既然mask和mask_tea都是同一个mask,还有必要在train里写入tensorboard对比吗?还是说只是占个位 -
flownet.py的FlownetCas中
for i in range(5):
if flow is not None:
flow_d, mask, feat = stu[i](torch.cat((warped_img0, warped_img1, warped_f0, warped_f1, timestep, mask, feat), 1), flow, scale=scale[i])
flow = flow + flow_d
else:
flow, mask, feat = stu[i](torch.cat((img0, img1, f0, f1, timestep), 1), None, scale=scale[i])
mask_list.append(mask)
flow_list.append(flow)
feat_list.append(feat[:, :1])
后续只用了feat第一层的置信度做融合,后面的层不是浪费了吗
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read dataset.py lines 64-67, model.py Model.update, and flownet.py FlownetCas; reproduce the fine-tuning with inference_video and compare training loss with validation LPIPS. The issue is complete when each reported behavior is explained and any confirmed code or training problem has a scoped fix and verification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100