huggingface / huggingface/diffusers

SDXL: enable_attention_slicing() + enable_model_cpu_offload() produces all-black images on MPS (Apple Silicon)

Open
#14,438 4 comments 0 reactions 0 assignees View on GitHub
bug pipelines
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

# SDXL: `enable_attention_slicing()` + `enable_model_cpu_offload()` produces all-black images on MPS

## Describe the bug

When using `enable_model_cpu_offload()` **combined with** `enable_attention_slicing()` on SDXL with Apple Silicon (MPS backend), the pipeline produces all-black output images due to NaN in the UNet attention layers. Removing `enable_attention_slicing()` produces valid output.

`enable_sequential_cpu_offload()` with attention slicing works correctly.

## Root Cause

The `SlicedAttnProcessor` produces NaN when the UNet is loaded via `model_cpu_offload` hooks on MPS. Attention slices appear to operate on uninitialized/stale memory regions. The NaN propagates through the UNet latents, and the VAE faithfully decodes corrupted latents to zeros (black).

**The VAE is not the failure point** — the NaN originates in the UNet during denoising.

## Isolation Results

| Configuration | Result |
|---|---|
| `model_cpu_offload()` without `enable_attention_slicing()` | ✅ Valid image (pixel mean=140.73) |
| `model_cpu_offload()` with `enable_attention_slicing()` | ❌ All black (pixel mean=0.0, NaN) |
| `sequential_cpu_offload()` with `enable_attention_slicing()` | ✅ Valid image (pixel mean=140.72) |
| `sequential_cpu_offload()` without `enable_attention_slicing()` | ✅ Valid image |

The bug is:
- **Prompt-agnostic** (anime, photorealistic, landscape, abstract — all fail)
- **Resolution-agnostic** (512×512 and 768×768 both fail)
- **Seed-agnostic** (tested seeds 42, 123, 7777)
- **Scheduler-agnostic** (Euler Ancestral, DPM++ 2M both fail)
- **VAE-agnostic** (both `madebyollin/sdxl-vae-fp16-fix` and default SDXL VAE)
- **Hardware-agnostic** (reproduced on M1 8GB and M5 Pro 24GB — not memory-pressure-related)

## Reproduction

```python
import torch
from diffusers import StableDiffusionXLPipeline, EulerAncestralDiscreteScheduler, AutoencoderKL

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)

pipe = StableDiffusionXLPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
vae=vae,
torch_dtype=torch.float16,
variant="fp16",
use_safetensors=True,
)
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()
pipe.enable_attention_slicing() # ← THIS is the trigger. Remove this line and output is valid.

result = pipe(
prompt="1girl, anime, walking through cherry blossom park, high quality, masterpiece",
negative_prompt="ugly, deformed, blurry, low quality, realistic",
num_inference_steps=20,
guidance_scale=7.5,
width=512,
height=512,
generator=torch.Generator("cpu").manual_seed(42),
)

import numpy as np
img_array = np.array(result.images[0])
print(f"Pixel mean: {img_array.mean()}") # 0.0
print(f"Pixel max: {img_array.max()}") # 0
```

**Expected:** Valid image with pixel values in [0, 255].
**Actual:** All-zero (black) image.

## Workaround

```python
pipe.enable_model_cpu_offload()
# pipe.enable_attention_slicing() # Do not use with model_cpu_offload on MPS
```

## Environment

| Component | Version |
|-----------|---------|
| OS | macOS (Apple Silicon) |
| Hardware | Apple M1 8GB / M5 Pro 24GB (both reproduce) |
| Python | 3.13.7 |
| PyTorch | 2.13.0 |
| diffusers | 0.39.0 |
| transformers | 5.14.1 |
| accelerate | 1.14.0 |

## Who can help?

@yiyixuxu @sayakpaul @DN6

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the provided SDXL script on MPS, then inspect SlicedAttnProcessor together with the model_cpu_offload hooks. Compare it with sequential CPU offload and unsliced attention; done means the combined configuration produces a valid non-black image without NaNs while the existing working configurations remain valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.