modelscope / modelscope/DiffSynth-Studio

wan训练时vae encode错误

Open
#1,113 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.1k
Forks
1.3k
Avg merge
13h 12m
Merged PRs (30d)
45

Description

Traceback (most recent call last):
File "train_wan_video.py", line 268, in
launcher_map[args.task](accelerator, dataset, model, model_logger, args=args)
File "diffsynth/diffusion/runner.py", line 34, in launch_training_task
loss = model(data)
File "train_wan_video.py", line 130, in forward
inputs = self.pipe.unit_runner(unit, self.pipe, *inputs)
File "diffsynth/pipelines/wan_video.py", line 102, in process
input_latents = pipe.vae.encode(input_video, device=pipe.device, tiled=tiled, ...)
File "diffsynth/models/wan_video_vae.py", line 512, in encode
hidden_state = self.single_encode(video, device)
File "diffsynth/models/wan_video_vae.py", line 545, in single_encode
# Here i=0, but self._enc_feat_map contains data from the PREVIOUS batch!
out = self.encoder(x, feat_cache=self._enc_feat_map, feat_idx=self._enc_conv_idx)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "diffsynth/models/wan_video_vae.py", line 980, in forward
# Inside Encoder Loop
x = layer(x, feat_cache, feat_idx)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "diffsynth/models/wan_video_vae.py", line 1150, in forward
# Inside ResBlock or DownSample Block
x = layer(x, feat_cache[idx])
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "diffsynth/models/wan_video_vae.py", line 1228, in forward
# CRASH HAPPENS HERE
# cache_x (from dirty cache) has 384 channels (Deep Feature)
# x (current input) has 96 channels (Shallow Feature)
x = torch.cat([cache_x, x], dim=2)
RuntimeError: sizes of tensors must match except in dimension 2. Expected size 384 but got size 96 for tensor number 1 in the list.

我训练wan2.1的时候发生了这个错误,我已经看过输入的input_video了是(1,3,121,480,832),应该没有问题,更进一步,我单独写了一个debug脚本测试了单独加载数据集和调用vae encode,,结果跑通了,没有出现问题,这是为什么,该怎么解决呢

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with train_wan_video.py and trace the training path through diffsynth/pipelines/wan_video.py into diffsynth/models/wan_video_vae.py, especially encode, single_encode, and the encoder cache arguments. Reproduce the full training call rather than the standalone dataset/VAE test, then compare cache state between batches and confirm the channel mismatch no longer occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.