kohya-ss / kohya-ss/sd-scripts

SD3.5 finetuning crashes due to dtype difference in vae call when not caching latents (potentially DDP related?)

Open
#1,758 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

```
[rank1]: Traceback (most recent call last):
[rank1]: File "/dockercontainer/sd-scripts/sd3_train.py", line 1200, in
[rank1]: train(args)
[rank1]: File "/dockercontainer/sd-scripts/sd3_train.py", line 871, in train
[rank1]: latents = vae.encode(batch["images"])
[rank1]: File "/dockercontainer/sd-scripts/library/sd3_models.py", line 1435, in encode
[rank1]: hidden = self.encoder(image)
[rank1]: File "/dockercontainer/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank1]: return self._call_impl(*args, **kwargs)
[rank1]: File "/dockercontainer/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank1]: return forward_call(*args, **kwargs)
[rank1]: File "/dockercontainer/sd-scripts/library/sd3_models.py", line 1333, in forward
[rank1]: hs = [self.conv_in(x)]
[rank1]: File "/dockercontainer/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank1]: return self._call_impl(*args, **kwargs)
[rank1]: File "/dockercontainer/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank1]: return forward_call(*args, **kwargs)
[rank1]: File "/dockercontainer/venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 554, in forward
[rank1]: return self._conv_forward(input, self.weight, self.bias)
[rank1]: File "/dockercontainer/venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 549, in _conv_forward
[rank1]: return F.conv2d(
RuntimeError: Input type (float) and bias type (c10::BFloat16) should be the same
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce SD3.5 finetuning without caching latents and start at the vae.encode call in sd3_train.py:871. Trace the input and model dtypes through library/sd3_models.py, especially encode at line 1435 and the encoder convolution at line 1333. Done means this path no longer raises the float versus BFloat16 mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.