huggingface / huggingface/diffusers
SDXL: enable_attention_slicing() + enable_model_cpu_offload() produces all-black images on MPS (Apple Silicon)
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
# SDXL: `enable_attention_slicing()` + `enable_model_cpu_offload()` produces all-black images on MPS
## Describe the bug
When using `enable_model_cpu_offload()` **combined with** `enable_attention_slicing()` on SDXL with Apple Silicon (MPS backend), the pipeline produces all-black output images due to NaN in the UNet attention layers. Removing `enable_attention_slicing()` produces valid output.
`enable_sequential_cpu_offload()` with attention slicing works correctly.
## Root Cause
The `SlicedAttnProcessor` produces NaN when the UNet is loaded via `model_cpu_offload` hooks on MPS. Attention slices appear to operate on uninitialized/stale memory regions. The NaN propagates through the UNet latents, and the VAE faithfully decodes corrupted latents to zeros (black).
**The VAE is not the failure point** — the NaN originates in the UNet during denoising.
## Isolation Results
| Configuration | Result |
|---|---|
| `model_cpu_offload()` without `enable_attention_slicing()` | ✅ Valid image (pixel mean=140.73) |
| `model_cpu_offload()` with `enable_attention_slicing()` | ❌ All black (pixel mean=0.0, NaN) |
| `sequential_cpu_offload()` with `enable_attention_slicing()` | ✅ Valid image (pixel mean=140.72) |
| `sequential_cpu_offload()` without `enable_attention_slicing()` | ✅ Valid image |
The bug is:
- **Prompt-agnostic** (anime, photorealistic, landscape, abstract — all fail)
- **Resolution-agnostic** (512×512 and 768×768 both fail)
- **Seed-agnostic** (tested seeds 42, 123, 7777)
- **Scheduler-agnostic** (Euler Ancestral, DPM++ 2M both fail)
- **VAE-agnostic** (both `madebyollin/sdxl-vae-fp16-fix` and default SDXL VAE)
- **Hardware-agnostic** (reproduced on M1 8GB and M5 Pro 24GB — not memory-pressure-related)
## Reproduction
```python
import torch
from diffusers import StableDiffusionXLPipeline, EulerAncestralDiscreteScheduler, AutoencoderKL
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
vae=vae,
torch_dtype=torch.float16,
variant="fp16",
use_safetensors=True,
)
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()
pipe.enable_attention_slicing() # ← THIS is the trigger. Remove this line and output is valid.
result = pipe(
prompt="1girl, anime, walking through cherry blossom park, high quality, masterpiece",
negative_prompt="ugly, deformed, blurry, low quality, realistic",
num_inference_steps=20,
guidance_scale=7.5,
width=512,
height=512,
generator=torch.Generator("cpu").manual_seed(42),
)
import numpy as np
img_array = np.array(result.images[0])
print(f"Pixel mean: {img_array.mean()}") # 0.0
print(f"Pixel max: {img_array.max()}") # 0
```
**Expected:** Valid image with pixel values in [0, 255].
**Actual:** All-zero (black) image.
## Workaround
```python
pipe.enable_model_cpu_offload()
# pipe.enable_attention_slicing() # Do not use with model_cpu_offload on MPS
```
## Environment
| Component | Version |
|-----------|---------|
| OS | macOS (Apple Silicon) |
| Hardware | Apple M1 8GB / M5 Pro 24GB (both reproduce) |
| Python | 3.13.7 |
| PyTorch | 2.13.0 |
| diffusers | 0.39.0 |
| transformers | 5.14.1 |
| accelerate | 1.14.0 |
## Who can help?
@yiyixuxu @sayakpaul @DN6
Contributor guide
Research direction
Start by reproducing the provided SDXL script on MPS, then inspect SlicedAttnProcessor together with the model_cpu_offload hooks. Compare it with sequential CPU offload and unsliced attention; done means the combined configuration produces a valid non-black image without NaNs while the existing working configurations remain valid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100