OOM with MPS on MBP M3 Max 128GB when training specific size of images.

### Describe the bug

1. When training diffusers, a bug occurs causing out-of-memory (OOM) errors or kernel crashes when the input image size is set to 128x128.
2. Sizes like 64x64, 256x256, and 512x512 do not trigger any errors, and memory usage remains stable, allowing for normal usage.
3. I experimented by only changing the image size while using the pipeline class from diffusers, focusing on the example code in the repository. I haven't really tweaked any other parameters. The UNet's down block and up block were adopted directly from the tutorials on the diffusers' homepage.

### Reproduction

run the dreambooth inpainting train script https://github.com/huggingface/diffusers/blob/main/examples/research_projects/dreambooth_inpaint/train_dreambooth_inpaint.py with difference image size on MBP M3 Max using MPS backend

### Logs

_No response_

### System Info

➜ mura-scripts pipenv run diffusers-cli env

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `diffusers` version: 0.21.2
- Platform: macOS-14.2.1-arm64-arm-64bit
- Python version: 3.11.7
- PyTorch version (GPU?): 2.1.2 (False)
- Huggingface_hub version: 0.15.1
- Transformers version: 4.30.2
- Accelerate version: 0.23.0
- xFormers version: not installed
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help?

_No response_

Inizia da examples/research_projects/dreambooth_inpaint/train_dreambooth_inpaint.py e riproduci l’esecuzione 128x128 su macOS con il backend MPS, confrontandola con 64x64, 256x256 e 512x512. Controlla le versioni riportate di diffusers, Python e PyTorch e raccogli i log; il lavoro è completo quando è stato identificato il motivo per cui solo il caso 128x128 esaurisce la memoria o causa il crash del kernel e il comportamento dopo la correzione è stato convalidato.

Scritto dal modello di indicizzazione a partire dal testo della issue.