kohya-ss / kohya-ss/sd-scripts
New training broken on Kaggle due to DistributedDataParallel and torch.distributed.elastic.multiprocessing.api
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
I am trying to do multi gpu training on Kaggle
Previously it was working great
But after all these new changes I am getting below error
```
Traceback (most recent call last):
File "/kaggle/working/kohya_ss/sd-scripts/train_db.py", line 529, in
train(args)
File "/kaggle/working/kohya_ss/sd-scripts/train_db.py", line 343, in train
encoder_hidden_states = train_util.get_hidden_states(
File "/kaggle/working/kohya_ss/sd-scripts/library/train_util.py", line 4427, in get_hidden_states
encoder_hidden_states = text_encoder.text_model.final_layer_norm(encoder_hidden_states)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__
raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'DistributedDataParallel' object has no attribute 'text_model'Traceback (most recent call last):
File "/kaggle/working/kohya_ss/sd-scripts/train_db.py", line 529, in
train(args)
File "/kaggle/working/kohya_ss/sd-scripts/train_db.py", line 343, in train
encoder_hidden_states = train_util.get_hidden_states(
File "/kaggle/working/kohya_ss/sd-scripts/library/train_util.py", line 4427, in get_hidden_states
encoder_hidden_states = text_encoder.text_model.final_layer_norm(encoder_hidden_states)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__
raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'DistributedDataParallel' object has no attribute 'text_model'
steps: 0%| | 0/3000 [00:00
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
args.func(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1008, in launch_command
multi_gpu_launcher(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 666, in multi_gpu_launcher
distrib_run.run(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
```
train command like this
```
Executing command: "/opt/conda/bin/accelerate" launch
--dynamo_backend no --dynamo_mode default --gpu_ids 0,1
--mixed_precision no --multi_gpu --num_processes 2
--num_machines 1 --num_cpu_threads_per_process 4
"/kaggle/working/kohya_ss/sd-scripts/train_db.py"
--config_file "./outputs/tmpfiledbooth.toml"
--max_grad_norm=0.0 --no_half_vae
--ddp_timeout=10000000 --ddp_gradient_as_bucket_view
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Kaggle accelerate launch command and inspect train_db.py around lines 343 and 529, then library/train_util.py around line 4427. Trace how the text encoder becomes a DistributedDataParallel object and compare that with the expected text_model access. Done means multi-GPU training proceeds past the initial step without the reported AttributeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100