pipeline does not work with conversion to fp16 in the to cuda call

Hi,

i was using and doing some fine-tuning with the Pixart model.
and i found a problem that i could pin it down to:

this code works:

pipe2 = PixArtAlphaPipeline.from_pretrained("PixArt-alpha/PixArt-XL-2-512x512", torch_dtype=torch.float16)
pipe2.to("cuda")
image = pipe2(prompt, num_inference_steps=20).images[0]    

while this does not, it gives out just noise images :

pipe1 = PixArtAlphaPipeline.from_pretrained("PixArt-alpha/PixArt-XL-2-512x512")
pipe1.to("cuda", dtype=torch.float16)
image = pipe1(prompt, num_inference_steps=20).images[0]    

i have looked through the code, and i cannot find any issues so far.
has anyone encountered a similar problem or has tips where to look further?

i am on WSL Ubuntu, diffusers==0.27.2, torch==2.2.2

Start by reproducing the two PixArtAlphaPipeline snippets with diffusers 0.27.2 and torch 2.2.2 on WSL Ubuntu, comparing the pipeline state and generated outputs after each CUDA conversion. Done means identifying why the two conversion paths differ and making the dtype conversion produce valid, non-noise images.