Issues on Wan2.1 Fun V1.1 Controlnet Training

Hi, I've recently started trainnng Wan2.1 Fun V1.1 14B Controlnet(canny edge) with my 200k videos.

1. I extracted 3 canny videos (sparse, medium, dense canny) from each original RGB videos.
2. each videos have 121 frames(10~20 videos may have less than 121, like 101 or 91)
3. I used just original example/.../Wan 2.1 Fun V1.1 14B Controlnet.sh script with just change for dataset path
4. my GPU env is , 80GB H100 x 8 GPUs and training the model consumes 77GB of all GPUs.
5. I installed accelerate, deepspeed and I didn't touch deepspeed config. I just set accelerate config for distributed GPU training.

After training started, I found

1. training speed is super slow:> 1%|█▉ | 495/63437 [8:21:06<1053:49:57, 60.27s/it].... why?

can I speedup?

2. sometimes I met error , so I had to start training from scratch as it saves no weight until it reaches 1 epoch. For that I have to wait 20 days.. > so , I changed

dataset_repeat: from 100 to 1
num_epochs : from 2 to 200
is it ok?
and wan training code has no argument like save_iter(like saving at every 10000 iter). should I change the training code on my own??

3. after some training iterations(660 or 264), I got error :

1%|██▌ | 660/63437 [11:08:23<1059:35:40, 60.76s/it]

[rank4]: Traceback (most recent call last):
[rank4]:   File "/home1/jovyan/workspace-2tb/DiffSynth-Studio/examples/wanvideo/model_training/train.py", line 131, in <module>
[rank4]:     launch_training_task(
[rank4]:   File "/home1/jovyan/workspace-2tb/DiffSynth-Studio/diffsynth/trainers/utils.py", line 402, in launch_training_task
[rank4]:     loss = model(data)
[rank4]:   File "/opt/conda/envs/wan22/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank4]:     return self._call_impl(*args, **kwargs)
[rank4]:   File "/opt/conda/envs/wan22/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank4]:     return forward_call(*args, **kwargs)
[rank4]:   File "/opt/conda/envs/wan22/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 20, in wrapped_fn
[rank4]:     ret_val = func(*args, **kwargs)
[rank4]:   File "/opt/conda/envs/wan22/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2105, in forward
[rank4]:     loss = self.module(*inputs, **kwargs)
[rank4]:   File "/opt/conda/envs/wan22/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank4]:     return self._call_impl(*args, **kwargs)
[rank4]:   File "/opt/conda/envs/wan22/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1857, in _call_impl
[rank4]:     return inner()
[rank4]:   File "/opt/conda/envs/wan22/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1805, in inner
[rank4]:     result = forward_call(*args, **kwargs)
[rank4]:   File "/home1/jovyan/workspace-2tb/DiffSynth-Studio/examples/wanvideo/model_training/train.py", line 97, in forward
[rank4]:     if inputs is None: inputs = self.forward_preprocess(data)
[rank4]:   File "/home1/jovyan/workspace-2tb/DiffSynth-Studio/examples/wanvideo/model_training/train.py", line 92, in forward_preprocess
[rank4]:     inputs_shared, inputs_posi, inputs_nega = self.pipe.unit_runner(unit, self.pipe, inputs_shared, inputs_posi, inputs_nega)
[rank4]:   File "/home1/jovyan/workspace-2tb/DiffSynth-Studio/diffsynth/utils/__init__.py", line 259, in __call__
[rank4]:     processor_outputs = unit.process(pipe, **processor_inputs)
[rank4]:   File "/home1/jovyan/workspace-2tb/DiffSynth-Studio/diffsynth/pipelines/wan_video_new.py", line 682, in process
[rank4]:     y = torch.concat([control_latents, y], dim=1)
[rank4]: RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 17 but got size 21 for tensor number 1 in the list.

Please help me~!

Start with examples/wanvideo/model_training/train.py and the Wan 2.1 Fun V1.1 Controlnet script, then trace the failing call into diffsynth/pipelines/wan_video_new.py:682 and the dataset frame handling. Reproduce the tensor-size error and measure the reported training behavior; done would require a confirmed cause, reproducible scope, and an agreed solution for the shape mismatch, checkpointing, or performance questions.