Multi GPU Training Fails - RuntimeErrordist._broadcast_coalesced(: Invalid scalar type - RuntimeError: Invalid scalar type

I have a subscriber who has dual RTX 4060 Ti - 16 GB

He is on Windows 10 and Python 3.10.9 - fresh install

When we set the huggingface default_config.yaml like below

```
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: MULTI_GPU
downcast_bf16: 'no'
gpu_ids: all
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

train util.py like below

```
accelerator = Accelerator(
gradient_accumulation_steps=args.gradient_accumulation_steps,
mixed_precision=args.mixed_precision,
log_with=log_with,
project_dir=logging_dir,
kwargs_handlers=[InitProcessGroupKwargs(backend="gloo")],
)
```

We are getting the below error. How can we fix it?

```
--ddp_gradient_as_bucket_view
NOTE: Redirects are currently not supported in Windows or MacOs.
[W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [pc]:29500 (system error: 10049 - unknown error).
[W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [pc]:29500 (system error: 10049 - unknown error).
prepare tokenizersprepare tokenizers

Using DreamBooth method.
Using DreamBooth method.
prepare images.
found directory C:\Users\user\Desktop\model\img\40_ohwx man contains 50 image files
No caption file found for 50 images. Training will continue without captions for these images. If class token exists, it will be used. / 50枚の画像にキャプションファイルが見つかりませんでした。これらの画像についてはキャプションなしで学習を 続行します。class tokenが存在する場合はそれを使います。
C:\Users\user\Desktop\model\img\40_ohwx man\man_10001.jpg
C:\Users\user\Desktop\model\img\40_ohwx man\man_10002.jpg
C:\Users\user\Desktop\model\img\40_ohwx man\man_10003.jpg
C:\Users\user\Desktop\model\img\40_ohwx man\man_10004.jpg
C:\Users\user\Desktop\model\img\40_ohwx man\man_10005.jpg
C:\Users\user\Desktop\model\img\40_ohwx man\man_10006.jpg... and 45 more
2000 train images with repeating.
0 reg images.
no regularization images / 正則化画像が見つかりませんでした
[Dataset 0]
batch_size: 1
resolution: (1024, 1024)
enable_bucket: False

[Subset 0 of Dataset 0]
image_dir: "C:\Users\user\Desktop\model\img\40_ohwx man"
image_count: 50
num_repeats: 40
shuffle_caption: False
keep_tokens: 0
keep_tokens_separator:
caption_dropout_rate: 0.0
caption_dropout_every_n_epoches: 0
caption_tag_dropout_rate: 0.0
caption_prefix: None
caption_suffix: None
color_aug: False
flip_aug: False
face_crop_aug_range: None
random_crop: False
token_warmup_min: 1,
token_warmup_step: 0,
is_reg: False
class_tokens: ohwx man
caption_extension: .caption

[Dataset 0]
loading image sizes.
100%|████████████████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 1612.82it/s]
prepare dataset
prepare accelerator
[W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [pc]:29500 (system error: 10049 - unknown error).
[W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [pc]:29500 (system error: 10049 - unknown error).
prepare images.
found directory C:\Users\user\Desktop\model\img\40_ohwx man contains 50 image files
No caption file found for 50 images. Training will continue without captions for these images. If class token exists, it will be used. / 50枚の画像にキャプションファイルが見つかりませんでした。これらの画像についてはキャプションなしで学習を 続行します。class tokenが存在する場合はそれを使います。
C:\Users\user\Desktop\model\img\40_ohwx man\man_10001.jpg
C:\Users\user\Desktop\model\img\40_ohwx man\man_10002.jpg
C:\Users\user\Desktop\model\img\40_ohwx man\man_10003.jpg
C:\Users\user\Desktop\model\img\40_ohwx man\man_10004.jpg
C:\Users\user\Desktop\model\img\40_ohwx man\man_10005.jpg
C:\Users\user\Desktop\model\img\40_ohwx man\man_10006.jpg... and 45 more
2000 train images with repeating.
0 reg images.
no regularization images / 正則化画像が見つかりませんでした
[Dataset 0]
batch_size: 1
resolution: (1024, 1024)
enable_bucket: False

[Subset 0 of Dataset 0]
image_dir: "C:\Users\user\Desktop\model\img\40_ohwx man"
image_count: 50
num_repeats: 40
shuffle_caption: False
keep_tokens: 0
keep_tokens_separator:
caption_dropout_rate: 0.0
caption_dropout_every_n_epoches: 0
caption_tag_dropout_rate: 0.0
caption_prefix: None
caption_suffix: None
color_aug: False
flip_aug: False
face_crop_aug_range: None
random_crop: False
token_warmup_min: 1,
token_warmup_step: 0,
is_reg: False
class_tokens: ohwx man
caption_extension: .caption

[Dataset 0]
loading image sizes.
100%|████████████████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 6247.66it/s]
prepare dataset
prepare accelerator
[W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [pc]:29500 (system error: 10049 - unknown error).
[W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [pc]:29500 (system error: 10049 - unknown error).
loading model for process 0/2
load StableDiffusion checkpoint: C:/Users/user/Downloads/sd_xl_base_1.0.safetensors
building U-Net
loading U-Net from checkpoint
U-Net:
building text encoders
loading text encoders from checkpoint
text encoder 1:
text encoder 2:
building VAE
loading VAE from checkpoint
VAE:
load VAE: stabilityai/sdxl-vae
additional VAE loaded
loading model for process 1/2
load StableDiffusion checkpoint: C:/Users/user/Downloads/sd_xl_base_1.0.safetensors
building U-Net
loading U-Net from checkpoint
U-Net:
building text encoders
loading text encoders from checkpoint
text encoder 1:
text encoder 2:
building VAE
loading VAE from checkpoint
VAE:
load VAE: stabilityai/sdxl-vae
additional VAE loaded
Disable Diffusers' xformers
Enable xformers for U-NetEnable xformers for U-Net

A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
[Dataset 0]
caching latents.
checking cache validity...
0%| | 0/50 [00:00
Traceback (most recent call last):
File "C:\Users\user\kohya_ss\sdxl_train.py", line 782, in
train(args)
File "C:\Users\user\kohya_ss\sdxl_train.py", line 399, in train
train(args)
File "C:\Users\user\kohya_ss\sdxl_train.py", line 399, in train
unet = accelerator.prepare(unet)
File "C:\Users\user\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 1284, in prepare
unet = accelerator.prepare(unet)
File "C:\Users\user\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 1284, in prepare
result = tuple(
File "C:\Users\user\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 1285, in
result = tuple(
File "C:\Users\user\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 1285, in
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
File "C:\Users\user\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 1090, in _prepare_one
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
File "C:\Users\user\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 1090, in _prepare_one
return self.prepare_model(obj, device_placement=device_placement)
File "C:\Users\user\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 1429, in prepare_model
return self.prepare_model(obj, device_placement=device_placement)
File "C:\Users\user\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 1429, in prepare_model
model = torch.nn.parallel.DistributedDataParallel(
File "C:\Users\user\kohya_ss\venv\lib\site-packages\torch\nn\parallel\distributed.py", line 676, in __init__
model = torch.nn.parallel.DistributedDataParallel(
File "C:\Users\user\kohya_ss\venv\lib\site-packages\torch\nn\parallel\distributed.py", line 676, in __init__
_sync_module_states( _sync_module_states(

File "C:\Users\user\kohya_ss\venv\lib\site-packages\torch\distributed\utils.py", line 142, in _sync_module_states
File "C:\Users\user\kohya_ss\venv\lib\site-packages\torch\distributed\utils.py", line 142, in _sync_module_states
_sync_params_and_buffers(
File "C:\Users\user\kohya_ss\venv\lib\site-packages\torch\distributed\utils.py", line 160, in _sync_params_and_buffers
_sync_params_and_buffers(
dist._broadcast_coalesced( File "C:\Users\user\kohya_ss\venv\lib\site-packages\torch\distributed\utils.py", line 160, in _sync_params_and_buffers

RuntimeErrordist._broadcast_coalesced(: Invalid scalar type

RuntimeError: Invalid scalar type
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 8908) of binary: C:\Users\user\kohya_ss\venv\Scripts\python.exe
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\user\kohya_ss\venv\Scripts\accelerate.exe\__main__.py", line 7, in
sys.exit(main())
File "C:\Users\user\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
args.func(args)
File "C:\Users\user\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 977, in launch_command
multi_gpu_launcher(args)
File "C:\Users\user\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 646, in multi_gpu_launcher
distrib_run.run(args)
File "C:\Users\user\kohya_ss\venv\lib\site-packages\torch\distributed\run.py", line 785, in run
elastic_launch(
File "C:\Users\user\kohya_ss\venv\lib\site-packages\torch\distributed\launcher\api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "C:\Users\user\kohya_ss\venv\lib\site-packages\torch\distributed\launcher\api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
./sdxl_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2024-01-13_00:35:21
host : pc
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 2948)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-01-13_00:35:21
host : pc
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 8908)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

```

Start with sdxl_train.py around lines 399 and 782, where Accelerator.prepare invokes distributed model setup, and compare that path with the MULTI_GPU configuration shown in the report. Reproduce the Windows dual-GPU launch and isolate whether the failure occurs during PyTorch parameter synchronization or the bf16 setup. Done means the reported training command passes model preparation without RuntimeError: Invalid scalar type.

Written by the indexing model from the issue text.