kohya-ss / kohya-ss/sd-scripts
SD3.5 finetuning crashes due to dtype difference in vae call when not caching latents (potentially DDP related?)
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
```
[rank1]: Traceback (most recent call last):
[rank1]: File "/dockercontainer/sd-scripts/sd3_train.py", line 1200, in
[rank1]: train(args)
[rank1]: File "/dockercontainer/sd-scripts/sd3_train.py", line 871, in train
[rank1]: latents = vae.encode(batch["images"])
[rank1]: File "/dockercontainer/sd-scripts/library/sd3_models.py", line 1435, in encode
[rank1]: hidden = self.encoder(image)
[rank1]: File "/dockercontainer/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank1]: return self._call_impl(*args, **kwargs)
[rank1]: File "/dockercontainer/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank1]: return forward_call(*args, **kwargs)
[rank1]: File "/dockercontainer/sd-scripts/library/sd3_models.py", line 1333, in forward
[rank1]: hs = [self.conv_in(x)]
[rank1]: File "/dockercontainer/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank1]: return self._call_impl(*args, **kwargs)
[rank1]: File "/dockercontainer/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank1]: return forward_call(*args, **kwargs)
[rank1]: File "/dockercontainer/venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 554, in forward
[rank1]: return self._conv_forward(input, self.weight, self.bias)
[rank1]: File "/dockercontainer/venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 549, in _conv_forward
[rank1]: return F.conv2d(
RuntimeError: Input type (float) and bias type (c10::BFloat16) should be the same
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce SD3.5 finetuning without caching latents and start at the vae.encode call in sd3_train.py:871. Trace the input and model dtypes through library/sd3_models.py, especially encode at line 1435 and the encoder convolution at line 1333. Done means this path no longer raises the float versus BFloat16 mismatch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100