diffusers fails in PyTorch when generating image using stabilityai/stable-diffusion-3.5-large-turbo, black-forest-labs/FLUX.1-dev on CPU

Describe the bug

Trace for stabilityai/stable-diffusion-3.5-large-turbo:

Traceback (most recent call last):
  File "/disks/samsung-4TB-A/AI-models/from-hugging-face/stable-diffusion-3.5-large-turbo/run.py", line 15, in <module>
    image = pipe(prompt).images[0]
            ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3.py", line 1068, in __call__
    noise_pred = self.transformer(
                 ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/diffusers/utils/peft_utils.py", line 315, in wrapper
    result = forward_fn(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/diffusers/models/transformers/transformer_sd3.py", line 291, in forward
    hidden_states = self.pos_embed(hidden_states)  # takes care of adding positional embeddings too.
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/diffusers/models/embeddings.py", line 559, in forward
    latent = self.proj(latent)
             ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 553, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 548, in _conv_forward
    return F.conv2d(
           ^^^^^^^^^
RuntimeError: Input type (c10::Half) and bias type (float) should be the same

Trace for black-forest-labs/FLUX.1-dev:

Traceback (most recent call last):
  File "/disks/samsung-4TB-A/AI-models/from-hugging-face/FLUX.1-dev/run.py", line 14, in <module>
    image = pipe(prompt).images[0]
            ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/diffusers/pipelines/flux/pipeline_flux.py", line 949, in __call__
    noise_pred = self.transformer(
                 ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/diffusers/utils/peft_utils.py", line 315, in wrapper
    result = forward_fn(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/diffusers/models/transformers/transformer_flux.py", line 680, in forward
    hidden_states = self.x_embedder(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 134, in forward
    return F.linear(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: mat1 and mat2 must have the same dtype, but got BFloat16 and Float

Reproduction

For stabilityai/stable-diffusion-3.5-large-turbo:

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-3.5-large-turbo", dtype=torch.bfloat16, device_map="cpu")
pipe.reset_device_map()
pipe.to("cpu")

prompt = "Bicycle in the forest, 8k"
image = pipe(prompt).images[0]

For black-forest-labs/FLUX.1-dev:

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", device_map="cpu")
pipe.reset_device_map()
pipe.to("cpu")

prompt = "A beautiful house next to the beach, 8k"
image = pipe(prompt).images[0]

Logs

See above.

System Info

OS: FreeBSD 15 STABLE amd64
diffusers-0.37.0
pytorch-2.10.0
Python-3.11

Who can help?

No response

Start with the CPU reproductions and the reported entry points in diffusers/models/embeddings.py, diffusers/models/transformers/transformer_sd3.py, and diffusers/models/transformers/transformer_flux.py. Compare the dtype handling at the failing convolution and linear layers for both pipelines, then rerun the two examples on the listed FreeBSD, PyTorch, and diffusers versions. Done means both model generations complete on CPU without dtype mismatch errors.