stable_video_diffusion model/pipeline review

# `stable_video_diffusion` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

Files/categories reviewed: target pipeline/model files, lazy imports/top-level exports/dummy objects, fast and slow tests, docs, deprecation status, dtype/device/offload/callback behavior, config validation, and related video pipeline precedents. Fast and slow tests exist; no missing slow-test item. Existing coverage still skips batch consistency and fp16 inference in `tests/pipelines/stable_video_diffusion/test_stable_video_diffusion.py`.

Duplicate search status: searched GitHub issues and PRs for `stable_video_diffusion`, `StableVideoDiffusionPipeline`, `UNetSpatioTemporalConditionModel`, `return_dict`, tensor image/CLIP resize, guidance scale, custom latents dtype, callback tensor inputs, and tuple config validation. No exact duplicates found except the tensor-image finding is a remaining/related part of closed issue https://github.com/huggingface/diffusers/issues/6574 and merged PR https://github.com/huggingface/diffusers/pull/6999.

## Issue 1: `return_dict=False` returns the raw frames object, not a tuple

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_video_diffusion/pipeline_stable_video_diffusion.py#L631-L632

Problem:
The pipeline returns `frames` directly when `return_dict=False`. Diffusers pipeline convention is to return a tuple, even for single-output pipelines. Current tests index `[0]`, which hides the bug for batch size 1 because indexing the tensor/list returns the first video rather than the first output field.

Impact:
Callers expecting the standard tuple contract get a `torch.Tensor`, `np.ndarray`, or list directly. This breaks generic pipeline wrappers and makes `pipe(..., return_dict=False)[0]` mean “first batch element” instead of “frames output”.

Reproduction:
```python
import torch
from transformers import CLIPImageProcessor, CLIPVisionConfig, CLIPVisionModelWithProjection
from diffusers import AutoencoderKLTemporalDecoder, EulerDiscreteScheduler, StableVideoDiffusionPipeline, UNetSpatioTemporalConditionModel

def make_pipe():
unet = UNetSpatioTemporalConditionModel(
block_out_channels=(32, 64), layers_per_block=1, sample_size=32, in_channels=8, out_channels=4,
down_block_types=("CrossAttnDownBlockSpatioTemporal", "DownBlockSpatioTemporal"),
up_block_types=("UpBlockSpatioTemporal", "CrossAttnUpBlockSpatioTemporal"),
cross_attention_dim=32, num_attention_heads=8,
projection_class_embeddings_input_dim=96, addition_time_embed_dim=32,
)
vae = AutoencoderKLTemporalDecoder(block_out_channels=[32, 64], in_channels=3, out_channels=3, down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"], latent_channels=4)
image_encoder = CLIPVisionModelWithProjection(CLIPVisionConfig(hidden_size=32, projection_dim=32, num_hidden_layers=1, num_attention_heads=4, image_size=32, intermediate_size=37, patch_size=1))
pipe = StableVideoDiffusionPipeline(vae=vae, image_encoder=image_encoder, unet=unet, scheduler=EulerDiscreteScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear"), feature_extractor=CLIPImageProcessor(crop_size=32, size=32))
pipe.set_progress_bar_config(disable=True)
return pipe

out = make_pipe()(image=torch.rand(1, 3, 32, 32), height=32, width=32, num_frames=2, num_inference_steps=1, output_type="pt", return_dict=False)
print(type(out), isinstance(out, tuple), out.shape)
# Current: False torch.Size([1, 2, 3, 32, 32])
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan.py#L672-L675
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/pipeline_ltx.py#L849-L852

Suggested fix:
```python
if not return_dict:
return (frames,)
```

## Issue 2: Decreasing guidance scales can crash CFG batching

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_video_diffusion/pipeline_stable_video_diffusion.py#L497-L503
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_video_diffusion/pipeline_stable_video_diffusion.py#L569-L587

Problem:
Before image and conditioning tensors are duplicated for CFG, the pipeline sets `self._guidance_scale = max_guidance_scale`. If `min_guidance_scale > 1` but `max_guidance_scale <= 1`, conditioning is prepared without CFG duplication. Later, `self._guidance_scale` becomes the full per-frame tensor, `do_classifier_free_guidance` becomes true, and the denoising loop duplicates latents only. The next concat with non-duplicated `image_latents` fails on batch size.

Impact:
A valid decreasing guidance schedule, for example stronger first-frame guidance and no final-frame guidance, crashes at runtime.

Reproduction:
```python
# Reuse the make_pipe() definition from Issue 1.
import torch

pipe = make_pipe()
try:
pipe(
image=torch.rand(1, 3, 32, 32),
height=32,
width=32,
num_frames=2,
num_inference_steps=1,
output_type="pt",
min_guidance_scale=2.0,
max_guidance_scale=1.0,
)
except Exception as e:
print(type(e).__name__, str(e).split("\n")[0])
# RuntimeError Sizes of tensors must match except in dimension 2. Expected size 2 but got size 1 ...
```

Relevant precedent:
Merged PR https://github.com/huggingface/diffusers/pull/7143 fixed a related CFG disable regression for scalar `max_guidance_scale=1`, but it does not cover guidance ranges where only `min_guidance_scale` crosses the CFG threshold.

Suggested fix:
```python
self._guidance_scale = max(min_guidance_scale, max_guidance_scale)
```

## Issue 3: Tensor image inputs are not resized before CLIP encoding

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_video_diffusion/pipeline_stable_video_diffusion.py#L200-L221

Problem:
The CLIP resize path is inside `if not isinstance(image, torch.Tensor)`. PIL/list inputs are resized to CLIP resolution before `image_encoder`, but tensor inputs go directly into `CLIPImageProcessor(..., do_resize=False)` and then CLIP. A normal SVD tensor input at generation size, such as `[1, 3, 576, 1024]`, is therefore incompatible with the CLIP image encoder.

Impact:
The docstring allows tensor images in `[0, 1]`, but users must secretly pre-resize tensors to the image encoder size. This is inconsistent with PIL inputs and with the closed tensor-input bug history in issue #6574 / PR #6999.

Reproduction:
```python
# Reuse the make_pipe() definition from Issue 1.
import torch

pipe = make_pipe()
try:
pipe(
image=torch.rand(1, 3, 64, 64),
height=64,
width=64,
num_frames=2,
num_inference_steps=1,
output_type="pt",
)
except Exception as e:
print(type(e).__name__, str(e).split("\n")[0])
# ValueError Input image size (64*64) doesn't match model (32*32).
```

Relevant precedent:
https://github.com/huggingface/diffusers/issues/6574
https://github.com/huggingface/diffusers/pull/6999

Suggested fix:
```python
if not isinstance(image, torch.Tensor):
image = self.video_processor.pil_to_numpy(image)
image = self.video_processor.numpy_to_pt(image)

image = image * 2.0 - 1.0
image = _resize_with_antialiasing(image, (224, 224))
image = (image + 1.0) / 2.0
```

## Issue 4: Custom latents are moved to device but not cast to pipeline dtype

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_video_diffusion/pipeline_stable_video_diffusion.py#L358-L364

Problem:
Generated latents use `dtype=image_embeddings.dtype`, but user-provided `latents` only call `latents.to(device)`. In a half-precision pipeline, float32 custom latents promote the concatenated UNet input to float32, while UNet weights are float16.

Impact:
Supplying precomputed fp32 latents to an fp16 pipeline can crash with dtype mismatches instead of being normalized to the pipeline’s working dtype.

Reproduction:
```python
# Reuse the make_pipe() definition from Issue 1.
import torch

pipe = make_pipe().to(dtype=torch.float16)
latents = torch.randn(1, 2, 4, 16, 16, dtype=torch.float32)
try:
pipe(image=torch.rand(1, 3, 32, 32), height=32, width=32, num_frames=2, num_inference_steps=1, output_type="pt", latents=latents)
except Exception as e:
print(type(e).__name__, str(e).split("\n")[0])
# RuntimeError mat1 and mat2 must have the same dtype, but got Float and Half
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan.py#L337-L338
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/pipeline_ltx.py#L486-L487

Suggested fix:
```python
else:
latents = latents.to(device=device, dtype=dtype)
```

## Issue 5: Tuple config length validation is incomplete in the spatio-temporal UNet

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/unets/unet_spatio_temporal_condition.py#L117-L160
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/unets/unet_spatio_temporal_condition.py#L170-L180
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/unets/unet_spatio_temporal_condition.py#L219-L223

Problem:
`cross_attention_dim` is validated only when it is a `list`, not a `tuple`, even though tuples are accepted and tested. `transformer_layers_per_block` is expanded when it is an int, but non-int sequence lengths are not validated before indexed access.

Impact:
Bad configs fail with `IndexError: tuple index out of range` during construction instead of a clear config `ValueError`. Longer tuples can also silently carry unused entries.

Reproduction:
```python
from diffusers import UNetSpatioTemporalConditionModel

try:
UNetSpatioTemporalConditionModel(
block_out_channels=(32, 64), layers_per_block=1, sample_size=32, in_channels=8, out_channels=4,
down_block_types=("CrossAttnDownBlockSpatioTemporal", "DownBlockSpatioTemporal"),
up_block_types=("UpBlockSpatioTemporal", "CrossAttnUpBlockSpatioTemporal"),
cross_attention_dim=(32,), num_attention_heads=8,
projection_class_embeddings_input_dim=96, addition_time_embed_dim=32,
)
except Exception as e:
print(type(e).__name__, str(e))
# IndexError tuple index out of range
```

Relevant precedent:
The same initializer already validates `num_attention_heads` and `layers_per_block` sequence lengths before indexing:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/unets/unet_spatio_temporal_condition.py#L112-L124

Suggested fix:
```python
if not isinstance(cross_attention_dim, int) and len(cross_attention_dim) != len(down_block_types):
raise ValueError(
f"Must provide the same number of `cross_attention_dim` as `down_block_types`. "
f"`cross_attention_dim`: {cross_attention_dim}. `down_block_types`: {down_block_types}."
)

if not isinstance(transformer_layers_per_block, int) and len(transformer_layers_per_block) != len(down_block_types):
raise ValueError(
f"Must provide the same number of `transformer_layers_per_block` as `down_block_types`. "
f"`transformer_layers_per_block`: {transformer_layers_per_block}. `down_block_types`: {down_block_types}."
)
```

Start with src/diffusers/pipelines/stable_video_diffusion/pipeline_stable_video_diffusion.py and its return, guidance, image, and latent entry points, then inspect the related UNet initializer in src/diffusers/models/unets/unet_spatio_temporal_condition.py. Run tests/pipelines/stable_video_diffusion/test_stable_video_diffusion.py and add regression coverage for the reported pipeline cases and invalid tuple configurations. Done means the five reproductions follow the documented contracts without runtime or indexing errors.

索引モデルが issue の本文から書いたものです。