error while training lycoris for flux.1

[2024-10-09 13:04:03,510] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 2
[2024-10-09 13:04:13,703] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2024-10-09 13:04:13,706] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2024-10-09 13:04:13,706] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2024-10-09 13:04:13,765] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
[2024-10-09 13:04:13,765] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=
[2024-10-09 13:04:13,766] [INFO] [logging.py:96:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False
[2024-10-09 13:04:13,766] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer
[rank1]: Traceback (most recent call last):
[rank1]: File "/hy-tmp/sd-scripts/flux_train_network.py", line 519, in
[rank1]: trainer.train(args)
[rank1]: File "/hy-tmp/sd-scripts/train_network.py", line 595, in train
[rank1]: ds_model, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(
[rank1]: ^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1303, in prepare
[rank1]: result = self._prepare_deepspeed(*args)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1779, in _prepare_deepspeed
[rank1]: engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/__init__.py", line 193, in initialize
[rank1]: engine = DeepSpeedEngine(args=args,
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 313, in __init__
[rank1]: self._configure_optimizer(optimizer, model_parameters)
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1302, in _configure_optimizer
[rank1]: self.optimizer = self._configure_zero_optimizer(basic_optimizer)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1626, in _configure_zero_optimizer
[rank1]: optimizer = DeepSpeedZeroOptimizer_Stage3(
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 161, in __init__
[rank1]: print_rank_0(f"initialized {__class__.__name__} with args: {locals()}", force=False)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in __repr__
[rank1]: mod_str = repr(module)
[rank1]: ^^^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in __repr__
[rank1]: mod_str = repr(module)
[rank1]: ^^^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in __repr__
[rank1]: mod_str = repr(module)
[rank1]: ^^^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in __repr__
[rank1]: list_of_reprs = [repr(item) for item in self]
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in
[rank1]: list_of_reprs = [repr(item) for item in self]
[rank1]: ^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in __repr__
[rank1]: mod_str = repr(module)
[rank1]: ^^^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in __repr__
[rank1]: mod_str = repr(module)
[rank1]: ^^^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2546, in __repr__
[rank1]: extra_repr = self.extra_repr()
[rank1]: ^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 120, in extra_repr
[rank1]: return f'in_features={self.in_features}, out_features={self.out_features}, bias={self.bias is not None}'
[rank1]: ^^^^^^^^^
[rank1]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1729, in __getattr__
[rank1]: raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
[rank1]: AttributeError: 'Linear' object has no attribute 'bias'
[2024-10-09 13:04:13,942] [INFO] [utils.py:781:see_memory_usage] Stage 3 initialize beginning
[2024-10-09 13:04:13,942] [INFO] [utils.py:782:see_memory_usage] MA 22.17 GB Max_MA 22.17 GB CA 22.18 GB Max_CA 22 GB
[2024-10-09 13:04:13,943] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 88.27 GB, percent = 17.5%
[rank0]: Traceback (most recent call last):
[rank0]: File "/hy-tmp/sd-scripts/flux_train_network.py", line 519, in
[rank0]: trainer.train(args)
[rank0]: File "/hy-tmp/sd-scripts/train_network.py", line 595, in train
[rank0]: ds_model, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1303, in prepare
[rank0]: result = self._prepare_deepspeed(*args)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/accelerate/accelerator.py", line 1779, in _prepare_deepspeed
[rank0]: engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/__init__.py", line 193, in initialize
[rank0]: engine = DeepSpeedEngine(args=args,
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 313, in __init__
[rank0]: self._configure_optimizer(optimizer, model_parameters)
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1302, in _configure_optimizer
[rank0]: self.optimizer = self._configure_zero_optimizer(basic_optimizer)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1626, in _configure_zero_optimizer
[rank0]: optimizer = DeepSpeedZeroOptimizer_Stage3(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 161, in __init__
[rank0]: print_rank_0(f"initialized {__class__.__name__} with args: {locals()}", force=False)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in __repr__
[rank0]: mod_str = repr(module)
[rank0]: ^^^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in __repr__
[rank0]: mod_str = repr(module)
[rank0]: ^^^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in __repr__
[rank0]: mod_str = repr(module)
[rank0]: ^^^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in __repr__
[rank0]: list_of_reprs = [repr(item) for item in self]
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/container.py", line 332, in
[rank0]: list_of_reprs = [repr(item) for item in self]
[rank0]: ^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in __repr__
[rank0]: mod_str = repr(module)
[rank0]: ^^^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2552, in __repr__
[rank0]: mod_str = repr(module)
[rank0]: ^^^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2546, in __repr__
[rank0]: extra_repr = self.extra_repr()
[rank0]: ^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 120, in extra_repr
[rank0]: return f'in_features={self.in_features}, out_features={self.out_features}, bias={self.bias is not None}'
[rank0]: ^^^^^^^^^
[rank0]: File "/usr/local/miniconda3/envs/ko/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1729, in __getattr__
[rank0]: raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
[rank0]: AttributeError: 'Linear' object has no attribute 'bias'
W1009 13:04:17.034000 140314240078464 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 6826 closing signal SIGTERM
E1009 13:04:17.600000 140314240078464 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 6825) of binary: /usr/local/miniconda3/envs/ko/bin/python

here is my command:

accelerate launch --mixed_precision bf16 --num_cpu_threads_per_process 4 flux_train_network.py --pretrained_model_name_or_path /hy-tmp/flux/flux1-dev.safetensors --clip_l /hy-tmp/flux/clip_l.safetensors --t5xxl /hy-tmp/flux/t5xxl_fp16.safetensors --ae /hy-tmp/flux/ae.safetensors --cache_latents_to_disk --save_model_as safetensors --sdpa --persistent_data_loader_workers --max_data_loader_n_workers 2 --seed 42 --gradient_checkpointing --mixed_precision bf16 --save_precision bf16 --network_module lycoris.kohya --network_dim 32 --network_alpha 16 --network_args "conv_dim=32" "conv_alpha=16" "algo=full" "bypass_mode=false" --optimizer_type "AdamW" --learning_rate 1e-4 --full_bf16 --network_train_unet_only --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk --max_train_epochs 3 --save_every_n_epochs 1 --dataset_config anomaly_bearing.toml --output_dir /hy-tmp/out --output_name anomaly_bearing_lyco_mk1 --timestep_sampling sigmoid --model_prediction_type raw --guidance_scale 1.0 --loss_type l2 --lr_scheduler constant_with_warmup --max_grad_norm 0.0 --xformers --deepspeed --zero_stage=3 --offload_optimizer_device="cpu" --offload_param_device="cpu"

Reproduce the command and trace the failure from flux_train_network.py:519 through train_network.py:595 and accelerator.prepare. Inspect the DeepSpeed ZeRO stage-3 initialization and the torch.nn.Linear representation that raises the missing bias attribute; done means training proceeds past this initialization without the AttributeError.

Written by the indexing model from the issue text.