CogX fails on MacOS requesting a 10TB buffer.

### Describe the bug

Tried to run the THUDM/CogVideoX1.5-5B model using Diffusers from git (20th Nov, approx 8:30am GMT)
The script failed with
```
hidden_states = F.scaled_dot_product_attention(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Invalid buffer size: 10973.48 GB
```
While these are big models, I suspect that 10TB of Ram is not being used by the CUDA users out there :-)

### Reproduction

```py
import torch
from diffusers import CogVideoXPipeline
from diffusers.utils import export_to_video

torch.mps.set_per_process_memory_fraction(0.0)

prompt = "A panda, dressed in a small, red jacket and a tiny hat, sits on a wooden stool in a serene bamboo forest. The panda's fluffy paws strum a miniature acoustic guitar, producing soft, melodic tunes. Nearby, a few other pandas gather, watching curiously and some clapping in rhythm. Sunlight filters through the tall bamboo, casting a gentle glow on the scene. The panda's face is expressive, showing concentration and joy as it plays. The background includes a small, flowing stream and vibrant green foliage, enhancing the peaceful and magical atmosphere of this unique musical performance."

pipe = CogVideoXPipeline.from_pretrained(
"THUDM/CogVideoX1.5-5B",
torch_dtype=torch.bfloat16
).to("mps")

#pipe.enable_sequential_cpu_offload()
pipe.vae.enable_tiling()
pipe.vae.enable_slicing()

video = pipe(
prompt=prompt,
num_videos_per_prompt=1,
num_inference_steps=50,
num_frames=81,
guidance_scale=6,
generator=torch.Generator(device="mps").manual_seed(42),
).frames[0]

export_to_video(video, "output.mp4", fps=8)
```

### Logs

```shell
The full output was

$ python cogx.py
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████| 4/4 [00:37<00:00, 9.25s/it]
Loading pipeline components...: 100%|█████████████████████████████████████████████████████| 5/5 [00:39<00:00, 7.91s/it]
0%| | 0/50 [00:18
video = pipe(
^^^^^
File "/Volumes/SSD2TB/AI/cog/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2TB/AI/cog/lib/python3.11/site-packages/diffusers/pipelines/cogvideo/pipeline_cogvideox.py", line 710, in __call__
noise_pred = self.transformer(
^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2TB/AI/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1740, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2TB/AI/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2TB/AI/cog/lib/python3.11/site-packages/diffusers/models/transformers/cogvideox_transformer_3d.py", line 503, in forward
hidden_states, encoder_hidden_states = block(
^^^^^^
File "/Volumes/SSD2TB/AI/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1740, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2TB/AI/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2TB/AI/cog/lib/python3.11/site-packages/diffusers/models/transformers/cogvideox_transformer_3d.py", line 132, in forward
attn_hidden_states, attn_encoder_hidden_states = self.attn1(
^^^^^^^^^^^
File "/Volumes/SSD2TB/AI/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1740, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2TB/AI/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2TB/AI/cog/lib/python3.11/site-packages/diffusers/models/attention_processor.py", line 530, in forward
return self.processor(
^^^^^^^^^^^^^^^
File "/Volumes/SSD2TB/AI/cog/lib/python3.11/site-packages/diffusers/models/attention_processor.py", line 2297, in __call__
hidden_states = F.scaled_dot_product_attention(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Invalid buffer size: 10973.48 GB
```
```

### System Info

- 🤗 Diffusers version: 0.32.0.dev0
- Platform: macOS-15.1.1-arm64-arm-64bit
- Running on Google Colab?: No
- Python version: 3.11.10
- PyTorch version (GPU?): 2.6.0.dev20241115 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.26.2
- Transformers version: 4.46.2
- Accelerate version: 1.1.1
- PEFT version: not installed
- Bitsandbytes version: not installed
- Safetensors version: 0.4.5
- xFormers version: not installed
- Accelerator: Apple M3
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@pcuenca

Reproduzca el fallo con el script de CogVideoXPipeline proporcionado en un Apple M3 y, después, inspeccione la llamada de attention en diffusers/models/attention_processor.py y su invocador en diffusers/models/transformers/cogvideox_transformer_3d.py. Trace por qué la ruta de MPS solicita un búfer de 10973.48 GB y verifique que la misma reproducción ya no falla con un error de tamaño de búfer no válido.

Escrito por el modelo de indexación a partir del texto del issue.