kohya-ss / kohya-ss/sd-scripts

New training broken on Kaggle due to DistributedDataParallel and torch.distributed.elastic.multiprocessing.api

Open
#1,272 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

I am trying to do multi gpu training on Kaggle

Previously it was working great

But after all these new changes I am getting below error

```
Traceback (most recent call last):
File "/kaggle/working/kohya_ss/sd-scripts/train_db.py", line 529, in
train(args)
File "/kaggle/working/kohya_ss/sd-scripts/train_db.py", line 343, in train
encoder_hidden_states = train_util.get_hidden_states(
File "/kaggle/working/kohya_ss/sd-scripts/library/train_util.py", line 4427, in get_hidden_states
encoder_hidden_states = text_encoder.text_model.final_layer_norm(encoder_hidden_states)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__
raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'DistributedDataParallel' object has no attribute 'text_model'Traceback (most recent call last):
File "/kaggle/working/kohya_ss/sd-scripts/train_db.py", line 529, in

train(args)
File "/kaggle/working/kohya_ss/sd-scripts/train_db.py", line 343, in train
encoder_hidden_states = train_util.get_hidden_states(
File "/kaggle/working/kohya_ss/sd-scripts/library/train_util.py", line 4427, in get_hidden_states
encoder_hidden_states = text_encoder.text_model.final_layer_norm(encoder_hidden_states)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__
raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'DistributedDataParallel' object has no attribute 'text_model'
steps: 0%| | 0/3000 [00:00
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
args.func(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1008, in launch_command
multi_gpu_launcher(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 666, in multi_gpu_launcher
distrib_run.run(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
```

train command like this

```
Executing command: "/opt/conda/bin/accelerate" launch
--dynamo_backend no --dynamo_mode default --gpu_ids 0,1
--mixed_precision no --multi_gpu --num_processes 2
--num_machines 1 --num_cpu_threads_per_process 4
"/kaggle/working/kohya_ss/sd-scripts/train_db.py"
--config_file "./outputs/tmpfiledbooth.toml"
--max_grad_norm=0.0 --no_half_vae
--ddp_timeout=10000000 --ddp_gradient_as_bucket_view
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Kaggle accelerate launch command and inspect train_db.py around lines 343 and 529, then library/train_util.py around line 4427. Trace how the text encoder becomes a DistributedDataParallel object and compare that with the expected text_model access. Done means multi-GPU training proceeds past the initial step without the reported AttributeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.