stable_diffusion model/pipeline review

stable_diffusion model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules. .ai/review-rules.md references AGENTS.md, but that file was absent in this checkout; all available referenced rule files were applied.

Duplicate search: searched GitHub Issues and PRs in huggingface/diffusers for stable_diffusion, affected class/function/file names, and failure modes. No duplicates found for Issues 1-7. Issue 8 has related coverage context in #11762 and #9371.

Issue 1: Stable Diffusion subpackage ONNX lazy imports are broken

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/__init__.py#L68-L83

Problem:
When ONNX is unavailable, diffusers.pipelines.stable_diffusion imports only dummy_onnx_objects, which contains OnnxRuntimeModel but not the Stable Diffusion ONNX pipeline dummies. Subpackage imports fail with ImportError instead of returning dummy classes with backend errors. The same block also advertises pipeline_onnx_stable_diffusion_inpaint_legacy, but that module does not exist under pipelines/stable_diffusion.

Impact:
Users importing from the public subpackage get inconsistent behavior compared with from diffusers import OnnxStableDiffusionPipeline. If ONNX is installed, the legacy lazy entry points at a nonexistent module.

Reproduction:

from diffusers.pipelines.stable_diffusion import OnnxStableDiffusionPipeline

Relevant precedent:
src/diffusers/pipelines/__init__.py uses dummy_torch_and_transformers_and_onnx_objects for these classes.

Suggested fix:

except OptionalDependencyNotAvailable:
    from ...utils import dummy_torch_and_transformers_and_onnx_objects
    _dummy_objects.update(get_objects_from_module(dummy_torch_and_transformers_and_onnx_objects))
else:
    ...
    # Drop this nonexistent stable_diffusion module entry, or route it through deprecated exports.
    # _import_structure["pipeline_onnx_stable_diffusion_inpaint_legacy"] = [...]

Issue 2: ONNX upscaler ignores user-provided latents

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_onnx_stable_diffusion_upscale.py#L470-L477

Problem:
OnnxStableDiffusionUpscalePipeline.__call__ accepts latents, but does not forward it to prepare_latents, so custom latents are silently discarded.

Impact:
Users cannot reproduce or edit generations with externally prepared latents, unlike the Torch upscaler.

Reproduction:

import types, numpy as np, torch
from types import SimpleNamespace
from diffusers.pipelines.stable_diffusion.pipeline_onnx_stable_diffusion_upscale import OnnxStableDiffusionUpscalePipeline

class Fake:
    config = SimpleNamespace(num_latent_channels=4, num_unet_input_channels=7)
    safety_checker = None
    def check_inputs(self, *a, **k): pass
    def _encode_prompt(self, *a, **k): return np.zeros((1, 77, 8), dtype=np.float32)

pipe = Fake()
custom = np.zeros((1, 4, 64, 64), dtype=np.float32)
def prepare_latents(self, *args, latents=None):
    assert latents is custom, f"latents was dropped: {latents!r}"
pipe.prepare_latents = types.MethodType(prepare_latents, pipe)

OnnxStableDiffusionUpscalePipeline.__call__(
    pipe, prompt="x", image=torch.zeros(1, 3, 64, 64),
    num_inference_steps=1, generator=np.random.RandomState(0), latents=custom,
)

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_upscale.py#L712-L721

Suggested fix:

latents = self.prepare_latents(
    batch_size * num_images_per_prompt,
    self.config.num_latent_channels,
    height,
    width,
    latents_dtype,
    generator,
    latents=latents,
)

Issue 3: ONNX upscaler crashes when classifier-free guidance is disabled

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_onnx_stable_diffusion_upscale.py#L543-L546

Problem:
noise_pred is recomputed from noise_pred_uncond and noise_pred_text outside the if do_classifier_free_guidance block. With guidance_scale <= 1.0, those variables are never assigned.

Impact:
OnnxStableDiffusionUpscalePipeline(..., guidance_scale=1.0) fails instead of running unconditional/no-CFG inference.

Reproduction:

import numpy as np, torch
from types import SimpleNamespace
from diffusers.pipelines.stable_diffusion.pipeline_onnx_stable_diffusion_upscale import OnnxStableDiffusionUpscalePipeline

class Bar:
    def __enter__(self): return self
    def __exit__(self, *a): pass
    def update(self): pass

class Scheduler:
    order = 1
    init_noise_sigma = 1.0
    def set_timesteps(self, n): self.timesteps = [np.array(1, dtype=np.float32)]
    def scale_model_input(self, sample, t): return sample

class Fake:
    config = SimpleNamespace(num_latent_channels=4, num_unet_input_channels=7)
    scheduler = Scheduler()
    low_res_scheduler = SimpleNamespace(add_noise=lambda image, noise, noise_level: image)
    unet = SimpleNamespace(
        model=SimpleNamespace(get_inputs=lambda: [SimpleNamespace(name="timestep", type="tensor(float)")]),
        __call__=lambda **kw: [np.zeros((1, 4, 64, 64), dtype=np.float32)],
    )
    safety_checker = None
    def check_inputs(self, *a, **k): pass
    def _encode_prompt(self, *a, **k): return np.zeros((1, 77, 8), dtype=np.float32)
    def prepare_latents(self, *a, **k): return np.zeros((1, 4, 64, 64), dtype=np.float32)
    def progress_bar(self, total): return Bar()

OnnxStableDiffusionUpscalePipeline.__call__(
    Fake(), prompt="x", image=torch.zeros(1, 3, 64, 64),
    num_inference_steps=1, guidance_scale=1.0, output_type="np",
)

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_upscale.py#L744-L747

Suggested fix:

if do_classifier_free_guidance:
    noise_pred_uncond, noise_pred_text = np.split(noise_pred, 2)
    noise_pred = noise_pred_uncond + guidance_scale * (noise_pred_text - noise_pred_uncond)

Issue 4: StableUnCLIPImageNormalizer breaks standard .to() kwargs

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/stable_unclip_image_normalizer.py#L40-L47

Problem:
The custom .to() signature accepts only torch_device and torch_dtype, so standard calls like .to(dtype=torch.float16) or .to(device="cuda") raise TypeError.

Impact:
This violates nn.Module/ModelMixin behavior and makes the component harder to use or move independently.

Reproduction:

import torch
from diffusers.pipelines.stable_diffusion.stable_unclip_image_normalizer import StableUnCLIPImageNormalizer

StableUnCLIPImageNormalizer().to(dtype=torch.float16)

Relevant precedent:
Other ModelMixin modules rely on inherited nn.Module.to.

Suggested fix:

# Remove the override entirely; registered Parameters move with nn.Module.to.
# Or, if keeping it:
def to(self, *args, **kwargs):
    return super().to(*args, **kwargs)

Issue 5: UNet QKV unfuse state is not safe

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/unets/unet_2d_condition.py#L821-L849

Problem:
original_attn_processors is not initialized in __init__, so unfuse_qkv_projections() before fuse_qkv_projections() raises AttributeError. Calling fuse_qkv_projections() twice also overwrites the saved original processors with fused processors, so unfuse_qkv_projections() cannot restore the original state.

Impact:
A public optimization API is not idempotent and can leave the model fused permanently in common enable-twice-then-disable flows.

Reproduction:

from diffusers import UNet2DConditionModel

model = UNet2DConditionModel(
    block_out_channels=(4, 8), norm_num_groups=4,
    down_block_types=("CrossAttnDownBlock2D", "DownBlock2D"),
    up_block_types=("UpBlock2D", "CrossAttnUpBlock2D"),
    cross_attention_dim=8, attention_head_dim=2,
    out_channels=4, in_channels=4, layers_per_block=1, sample_size=16,
)

model.unfuse_qkv_projections()  # AttributeError

model.fuse_qkv_projections()
model.fuse_qkv_projections()
model.unfuse_qkv_projections()
print({p.__class__.__name__ for p in model.attn_processors.values()})  # still fused

Relevant precedent:
The method doc says it disables fused projections “if enabled”.

Suggested fix:

# in __init__
self.original_attn_processors = None

# in fuse_qkv_projections
if self.original_attn_processors is None:
    self.original_attn_processors = self.attn_processors

# in unfuse_qkv_projections
if self.original_attn_processors is not None:
    self.set_attn_processor(self.original_attn_processors)
    self.original_attn_processors = None

Issue 6: Shorter UNet cross-attention masks crash instead of padding correctly

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/unets/unet_2d_condition.py#L1073-L1076
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/attention_processor.py#L729-L742

Problem:
UNet2DConditionModel accepts encoder_attention_mask, but a mask shorter than encoder_hidden_states crashes because prepare_attention_mask pads by target_length instead of the remaining length. The repo already has this case as a skipped test.

Impact:
Callers cannot pass shortened cross-attention masks even though the attention code has padding logic for mismatched mask lengths.

Reproduction:

import torch
from diffusers import UNet2DConditionModel

torch.manual_seed(0)
model = UNet2DConditionModel(
    block_out_channels=(4, 8), norm_num_groups=4,
    down_block_types=("CrossAttnDownBlock2D", "DownBlock2D"),
    up_block_types=("UpBlock2D", "CrossAttnUpBlock2D"),
    cross_attention_dim=8, attention_head_dim=2,
    out_channels=4, in_channels=4, layers_per_block=1, sample_size=16,
).eval()

sample = torch.randn(1, 4, 16, 16)
cond = torch.randn(1, 4, 8)
short_mask = torch.zeros(1, 3, dtype=torch.bool)

with torch.no_grad():
    model(sample, torch.tensor([10]), cond, encoder_attention_mask=short_mask)

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/models/unets/test_models_unet_2d_condition.py#L678-L705

Suggested fix:
The fix is slightly risky because comments mention UnCLIP compatibility. Add separate tests for SD cross-attn masks and UnCLIP added-KV masks, then pad by target_length - current_length for the SD cross-attn case rather than by target_length.

Issue 7: checkpoint conversion ignores local_files_only in some SD branches

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py#L1556-L1562
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py#L1588-L1594

Problem:
download_from_original_stable_diffusion_ckpt(..., local_files_only=True) still calls the x4 upscaler scheduler loaders without local_files_only, and the Stable UnCLIP img2img branch calls stable_unclip_image_encoder(original_config) without forwarding local_files_only.

Impact:
Offline/local-only conversion can unexpectedly hit the network or fail with less useful errors.

Reproduction:

import ast, inspect
import diffusers.pipelines.stable_diffusion.convert_from_ckpt as c

tree = ast.parse(inspect.getsource(c.download_from_original_stable_diffusion_ckpt))
calls = [n for n in ast.walk(tree) if isinstance(n, ast.Call)]
print(any(getattr(getattr(n, "func", None), "id", "") == "stable_unclip_image_encoder"
          and not any(k.arg == "local_files_only" for k in n.keywords) for n in calls))

Relevant precedent:
Nearby loaders in the same function pass local_files_only=local_files_only.

Suggested fix:

scheduler = DDIMScheduler.from_pretrained(
    "stabilityai/stable-diffusion-x4-upscaler", subfolder="scheduler", local_files_only=local_files_only
)
low_res_scheduler = DDPMScheduler.from_pretrained(
    "stabilityai/stable-diffusion-x4-upscaler", subfolder="low_res_scheduler", local_files_only=local_files_only
)

feature_extractor, image_encoder = stable_unclip_image_encoder(
    original_config, local_files_only=local_files_only
)

Issue 8: Missing or disabled slow coverage for parts of the target family

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/stable_unclip/test_stable_unclip.py#L202-L204
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/stable_unclip/test_stable_unclip_img2img.py#L219-L221
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/stable_diffusion/test_onnx_stable_diffusion_upscale.py#L45-L51

Problem:
Stable UnCLIP integration tests are @nightly but not @slow. Flax SD img2img/inpaint have no direct test files. ONNX upscaler fast tests are entirely skipped, and integration coverage is nightly-only.

Impact:
Release slow CI can miss regressions in these target pipelines. This likely allowed the ONNX upscaler no-CFG and ignored-latents regressions above to survive.

Reproduction:

from pathlib import Path

checks = {
    "stable_unclip": "tests/pipelines/stable_unclip/test_stable_unclip.py",
    "stable_unclip_img2img": "tests/pipelines/stable_unclip/test_stable_unclip_img2img.py",
    "onnx_upscale": "tests/pipelines/stable_diffusion/test_onnx_stable_diffusion_upscale.py",
}
for name, path in checks.items():
    text = Path(path).read_text()
    print(name, "@slow" in text, "@nightly" in text, "@unittest.skip" in text)

Relevant precedent:
Core Torch SD, img2img, inpaint, depth, latent-upscale, and x4-upscale have slow tests. Related existing issues: https://github.com/huggingface/diffusers/issues/11762 documents the risky ONNX upscaler checkpoint that led to skipped tests; https://github.com/huggingface/diffusers/issues/9371 is an open Flax img2img API cleanup request but does not cover slow coverage.

Suggested fix:
Add @slow integration tests for Stable UnCLIP and Stable UnCLIP img2img, add direct Flax img2img/inpaint coverage or explicitly deprecate those pipelines, and replace the skipped ONNX upscaler checkpoint with a safe internal tiny ONNX fixture that covers guidance_scale=1.0 and custom latents.

Local checks: utils/check_copies.py passed. Targeted pytest collection was blocked by this .venv Torch build missing torch._C._distributed_c10d when importing test utilities.

Treat the report as seven separate fixes and start with the affected files and reproductions listed for each issue, especially the Stable Diffusion init, ONNX upscaler, UNet, attention processor, and checkpoint conversion paths. Run the existing UNet test in tests/models/unets/test_models_unet_2d_condition.py and add focused coverage for the reported failures. Done means each reproduction works as described without regressing the referenced behavior.