huggingface / huggingface/diffusers

Model and input data type is not same

Open
#8,780 1 comment 0 reactions 0 assignees View on GitHub
stale
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

**Is your feature request related to a problem? Please describe.**
Hi, when I trained sdv1.5 model with fp16 mode by using the `examples/text_to_image/train_text_to_image.py` file, I found there is a mismatch between unet model and input data. Specificaly, In this [line](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image.py#L993) , the `unet` model has float32 dtype, but the `noisy_latents` has the float16 dtype. Although it will not raise an error in cuda , I use my custom device it will raise an error, I wonder how can I change this code to use float16.

**Describe the solution you'd like.**
To avoid get a wrong model, I would like you give a right code to match model and input.

**Describe alternatives you've considered.**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context.**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Read examples/text_to_image/train_text_to_image.py around line 993, focusing on the dtype of the unet model and noisy_latents in fp16 mode. Reproduce the mismatch on the reported custom device and verify that the training path passes matching dtypes without breaking the existing CUDA behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.