pipeline does not work with conversion to fp16 in the to cuda call

Hi,

i was using and doing some fine-tuning with the Pixart model.
and i found a problem that i could pin it down to:

this code works:
```
pipe2 = PixArtAlphaPipeline.from_pretrained("PixArt-alpha/PixArt-XL-2-512x512", torch_dtype=torch.float16)
pipe2.to("cuda")
image = pipe2(prompt, num_inference_steps=20).images[0]

```

while this does not, it gives out just noise images :
```
pipe1 = PixArtAlphaPipeline.from_pretrained("PixArt-alpha/PixArt-XL-2-512x512")
pipe1.to("cuda", dtype=torch.float16)
image = pipe1(prompt, num_inference_steps=20).images[0]
```

i have looked through the code, and i cannot find any issues so far.
has anyone encountered a similar problem or has tips where to look further?

i am on WSL Ubuntu, diffusers==0.27.2, torch==2.2.2