modelscope / modelscope/DiffSynth-Studio

fsdp with wan2.2

Open
#882 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.1k
Forks
1.3k
Avg merge
13h 12m
Merged PRs (30d)
45

Description

[rank1]: File "/home/lcq/lijiarui/DiffSynth-Studio/examples/wanvideo/model_training/train.py", line 165, in
[rank1]: launch_training_task(
[rank1]: File "/home/lcq/lijiarui/DiffSynth-Studio/diffsynth/trainers/utils.py", line 491, in launch_training_task
[rank1]: loss = model(data=None, inputs=data)
[rank1]: File "/root/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank1]: return self._call_impl(*args, **kwargs)
[rank1]: File "/root/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank1]: return forward_call(*args, **kwargs)
[rank1]: File "/home/lcq/lijiarui/DiffSynth-Studio/examples/wanvideo/model_training/train.py", line 111, in forward
[rank1]: loss = self.pipe.training_loss(**models, **inputs)
[rank1]: File "/home/lcq/lijiarui/DiffSynth-Studio/diffsynth/pipelines/wan_video_new.py", line 86, in training_loss
[rank1]: noise_pred = self.model_fn(**inputs, timestep=timestep)
[rank1]: File "/home/lcq/lijiarui/DiffSynth-Studio/diffsynth/pipelines/wan_video_new.py", line 1230, in model_fn_wan_video
[rank1]: x = torch.utils.checkpoint.checkpoint(
[rank1]: File "/root/miniconda3/envs/py310/lib/python3.10/site-packages/torch/_compile.py", line 32, in inner
[rank1]: return disable_fn(*args, **kwargs)
[rank1]: File "/root/miniconda3/envs/py310/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 632, in _fn
[rank1]: return fn(*args, **kwargs)
[rank1]: File "/root/miniconda3/envs/py310/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 496, in checkpoint
[rank1]: ret = function(*args, **kwargs)
[rank1]: File "/home/lcq/lijiarui/DiffSynth-Studio/diffsynth/pipelines/wan_video_new.py", line 1218, in custom_forward
[rank1]: return module(*inputs)
[rank1]: File "/root/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank1]: return self._call_impl(*args, **kwargs)
[rank1]: File "/root/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank1]: return forward_call(*args, **kwargs)
[rank1]: File "/home/lcq/lijiarui/DiffSynth-Studio/diffsynth/models/wan_video_dit.py", line 220, in forward
[rank1]: self.modulation.to(dtype=t_mod.dtype, device=t_mod.device) + t_mod).chunk(6, dim=chunk_dim)
[rank1]: RuntimeError: Output 0 of ViewBackward0 is a view and its base or another view of its base has been modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one.

my FSDP config is:
compute_environment: LOCAL_MACHINE
debug: false
fsdp_config:
fsdp_activation_checkpointing: false
fsdp_auto_wrap_policy: SIZE_BASED_WRAP
fsdp_backward_prefetch: BACKWARD_PRE
fsdp_cpu_ram_efficient_loading: false
fsdp_forward_prefetch: false
fsdp_min_num_params: 100000
fsdp_offload_params: false
fsdp_sharding_strategy: FULL_SHARD
fsdp_state_dict_type: SHARDED_STATE_DICT
fsdp_sync_module_states: false
fsdp_use_orig_params: false
distributed_type: FSDP
downcast_bf16: 'no'
enable_cpu_affinity: false
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure using the supplied FSDP configuration and the Wan training entry point in examples/wanvideo/model_training/train.py. Inspect diffsynth/pipelines/wan_video_new.py around model_fn_wan_video and diffsynth/models/wan_video_dit.py around line 220, where the traceback ends. Done means the training run no longer raises the inplace view error under FSDP.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.