OOM with MPS on MBP M3 Max 128GB when training specific size of images.

### Describe the bug

1. When training diffusers, a bug occurs causing out-of-memory (OOM) errors or kernel crashes when the input image size is set to 128x128.
2. Sizes like 64x64, 256x256, and 512x512 do not trigger any errors, and memory usage remains stable, allowing for normal usage.
3. I experimented by only changing the image size while using the pipeline class from diffusers, focusing on the example code in the repository. I haven't really tweaked any other parameters. The UNet's down block and up block were adopted directly from the tutorials on the diffusers' homepage.

### Reproduction

run the dreambooth inpainting train script https://github.com/huggingface/diffusers/blob/main/examples/research_projects/dreambooth_inpaint/train_dreambooth_inpaint.py with difference image size on MBP M3 Max using MPS backend

### Logs

_No response_

### System Info

➜ mura-scripts pipenv run diffusers-cli env

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `diffusers` version: 0.21.2
- Platform: macOS-14.2.1-arm64-arm-64bit
- Python version: 3.11.7
- PyTorch version (GPU?): 2.1.2 (False)
- Huggingface_hub version: 0.15.1
- Transformers version: 4.30.2
- Accelerate version: 0.23.0
- xFormers version: not installed
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help?

_No response_