`StableDiffusionPipeline` produces `nans` for SD2.1 (Mac, cpu device)

### Describe the bug

`StableDiffusionPipeline` produces `nans` for SD2.1.


The issue seems to be in the `pipe.unet.conv_in`. Printing the mean / std for `sample` before and after `conv_in` gives:
```
m,s = tensor(0.005) tensor(0.996)
m,s = tensor(15962061100191580160., grad_fn=) tensor(16510648845702789070848., grad_fn=)
```

### Reproduction

Make sure I'm using the latest official diffusers version
```
# uninstall local, editable version
!pip uninstall diffusers -y -qq
# install official version
!pip install diffusers -qq
# restart notebook now
```

Code to reproduce
```
from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained('stabilityai/stable-diffusion-2-1')
result = pipe('the sun', output_type='pt')
result.images[0]
```

Output:
```
tensor([[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...
```

### Logs

_No response_

### System Info

- `diffusers` version: 0.24.0
- Platform: macOS-14.1.2-arm64-arm-64bit
- Python version: 3.11.6
- PyTorch version (GPU?): 2.1.1 (False)
- Huggingface_hub version: 0.19.4
- Transformers version: 4.35.2
- Accelerate version: 0.25.0
- xFormers version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

Questions on pipelines > Stable Diffusion: @yiyixuxu @DN6 @sayakpaul @patrickvonplaten