stable_video_diffusion model/pipeline review

# `stable_video_diffusion` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

Files/categories reviewed: target pipeline/model files, lazy imports/top-level exports/dummy objects, fast and slow tests, docs, deprecation status, dtype/device/offload/callback behavior, config validation, and related video pipeline precedents. Fast and slow tests exist; no missing slow-test item. Existing coverage still skips batch consistency and fp16 inference in `tests/pipelines/stable_video_diffusion/test_stable_video_diffusion.py`.

Duplicate search status: searched GitHub issues and PRs for `stable_video_diffusion`, `StableVideoDiffusionPipeline`, `UNetSpatioTemporalConditionModel`, `return_dict`, tensor image/CLIP resize, guidance scale, custom latents dtype, callback tensor inputs, and tuple config validation. No exact duplicates found except the tensor-image finding is a remaining/related part of closed issue https://github.com/huggingface/diffusers/issues/6574 and merged PR https://github.com/huggingface/diffusers/pull/6999.

## Issue 1: `return_dict=False` returns the raw frames object, not a tuple

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_video_diffusion/pipeline_stable_video_diffusion.py#L631-L632

Problem:
The pipeline returns `frames` directly when `return_dict=False`. Diffusers pipeline convention is to return a tuple, even for single-output pipelines. Current tests index `[0]`, which hides the bug for batch size 1 because indexing the tensor/list returns the first video rather than the first output field.

Impact:
Callers expecting the standard tuple contract get a `torch.Tensor`, `np.ndarray`, or list directly. This breaks generic pipeline wrappers and makes `pipe(..., return_dict=False)[0]` mean “first batch element” instead of “frames output”.

Reproduction:
```python
import torch
from transformers import CLIPImageProcessor, CLIPVisionConfig, CLIPVisionModelWithProjection
from diffusers import AutoencoderKLTemporalDecoder, EulerDiscreteScheduler, StableVideoDiffusionPipeline, UNetSpatioTemporalConditionModel

def make_pipe():
unet = UNetSpatioTemporalConditionModel(
block_out_channels=(32, 64), layers_per_block=1, sample_size=32, in_channels=8, out_channels=4,
down_block_types=("CrossAttnDownBlockSpatioTemporal", "DownBlockSpatioTemporal"),
up_block_types=("UpBlockSpatioTemporal", "CrossAttnUpBlockSpatioTemporal"),
cross_attention_dim=32, num_attention_heads=8,
projection_class_embeddings_input_dim=96, addition_time_embed_dim=32,
)
vae = AutoencoderKLTemporalDecoder(block_out_channels=[32, 64], in_channels=3, out_channels=3, down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"], latent_channels=4)
image_encoder = CLIPVisionModelWithProjection(CLIPVisionConfig(hidden_size=32, projection_dim=32, num_hidden_layers=1, num_attention_heads=4, image_size=32, intermediate_size=37, patch_size=1))
pipe = StableVideoDiffusionPipeline(vae=vae, image_encoder=image_encoder, unet=unet, scheduler=EulerDiscreteScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear"), feature_extractor=CLIPImageProcessor(crop_size=32, size=32))
pipe.set_progress_bar_config(disable=True)
return pipe

out = make_pipe()(image=torch.rand(1, 3, 32, 32), height=32, width=32, num_frames=2, num_inference_steps=1, output_type="pt", return_dict=False)
print(type(out), isinstance(out, tuple), out.shape)
# Current: False torch.Size([1, 2, 3, 32, 32])
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan.py#L672-L675
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/pipeline_ltx.py#L849-L852

Suggested fix:
```python
if not return_dict:
return (frames,)
```

## Issue 2: Decreasing guidance scales can crash CFG batching

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_video_diffusion/pipeline_stable_video_diffusion.py#L497-L503
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_video_diffusion/pipeline_stable_video_diffusion.py#L569-L587

Problem:
Before image and conditioning tensors are duplicated for CFG, the pipeline sets `self._guidance_scale = max_guidance_scale`. If `min_guidance_scale > 1` but `max_guidance_scale <= 1`, conditioning is prepared without CFG duplication. Later, `self._guidance_scale` becomes the full per-frame tensor, `do_classifier_free_guidance` becomes true, and the denoising loop duplicates latents only. The next concat with non-duplicated `image_latents` fails on batch size.

Impact:
A valid decreasing guidance schedule, for example stronger first-frame guidance and no final-frame guidance, crashes at runtime.

Reproduction:
```python
# Reuse the make_pipe() definition from Issue 1.
import torch

pipe = make_pipe()
try:
pipe(
image=torch.rand(1, 3, 32, 32),
height=32,
width=32,
num_frames=2,
num_inference_steps=1,
output_type="pt",
min_guidance_scale=2.0,
max_guidance_scale=1.0,
)
except Exception as e:
print(type(e).__name__, str(e).split("\n")[0])
# RuntimeError Sizes of tensors must match except in dimension 2. Expected size 2 but got size 1 ...
```

Relevant precedent:
Merged PR https://github.com/huggingface/diffusers/pull/7143 fixed a related CFG disable regression for scalar `max_guidance_scale=1`, but it does not cover guidance ranges where only `min_guidance_scale` crosses the CFG threshold.

Suggested fix:
```python
self._guidance_scale = max(min_guidance_scale, max_guidance_scale)
```

## Issue 3: Tensor image inputs are not resized before CLIP encoding

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_video_diffusion/pipeline_stable_video_diffusion.py#L200-L221

Problem:
The CLIP resize path is inside `if not isinstance(image, torch.Tensor)`. PIL/list inputs are resized to CLIP resolution before `image_encoder`, but tensor inputs go directly into `CLIPImageProcessor(..., do_resize=False)` and then CLIP. A normal SVD tensor input at generation size, such as `[1, 3, 576, 1024]`, is therefore incompatible with the CLIP image encoder.

Impact:
The docstring allows tensor images in `[0, 1]`, but users must secretly pre-resize tensors to the image encoder size. This is inconsistent with PIL inputs and with the closed tensor-input bug history in issue #6574 / PR #6999.

Reproduction:
```python
# Reuse the make_pipe() definition from Issue 1.
import torch

pipe = make_pipe()
try:
pipe(
image=torch.rand(1, 3, 64, 64),
height=64,
width=64,
num_frames=2,
num_inference_steps=1,
output_type="pt",
)
except Exception as e:
print(type(e).__name__, str(e).split("\n")[0])
# ValueError Input image size (64*64) doesn't match model (32*32).
```

Relevant precedent:
https://github.com/huggingface/diffusers/issues/6574
https://github.com/huggingface/diffusers/pull/6999

Suggested fix:
```python
if not isinstance(image, torch.Tensor):
image = self.video_processor.pil_to_numpy(image)
image = self.video_processor.numpy_to_pt(image)

image = image * 2.0 - 1.0
image = _resize_with_antialiasing(image, (224, 224))
image = (image + 1.0) / 2.0
```

## Issue 4: Custom latents are moved to device but not cast to pipeline dtype

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_video_diffusion/pipeline_stable_video_diffusion.py#L358-L364

Problem:
Generated latents use `dtype=image_embeddings.dtype`, but user-provided `latents` only call `latents.to(device)`. In a half-precision pipeline, float32 custom latents promote the concatenated UNet input to float32, while UNet weights are float16.

Impact:
Supplying precomputed fp32 latents to an fp16 pipeline can crash with dtype mismatches instead of being normalized to the pipeline’s working dtype.

Reproduction:
```python
# Reuse the make_pipe() definition from Issue 1.
import torch

pipe = make_pipe().to(dtype=torch.float16)
latents = torch.randn(1, 2, 4, 16, 16, dtype=torch.float32)
try:
pipe(image=torch.rand(1, 3, 32, 32), height=32, width=32, num_frames=2, num_inference_steps=1, output_type="pt", latents=latents)
except Exception as e:
print(type(e).__name__, str(e).split("\n")[0])
# RuntimeError mat1 and mat2 must have the same dtype, but got Float and Half
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan.py#L337-L338
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/pipeline_ltx.py#L486-L487

Suggested fix:
```python
else:
latents = latents.to(device=device, dtype=dtype)
```

## Issue 5: Tuple config length validation is incomplete in the spatio-temporal UNet

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/unets/unet_spatio_temporal_condition.py#L117-L160
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/unets/unet_spatio_temporal_condition.py#L170-L180
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/unets/unet_spatio_temporal_condition.py#L219-L223

Problem:
`cross_attention_dim` is validated only when it is a `list`, not a `tuple`, even though tuples are accepted and tested. `transformer_layers_per_block` is expanded when it is an int, but non-int sequence lengths are not validated before indexed access.

Impact:
Bad configs fail with `IndexError: tuple index out of range` during construction instead of a clear config `ValueError`. Longer tuples can also silently carry unused entries.

Reproduction:
```python
from diffusers import UNetSpatioTemporalConditionModel

try:
UNetSpatioTemporalConditionModel(
block_out_channels=(32, 64), layers_per_block=1, sample_size=32, in_channels=8, out_channels=4,
down_block_types=("CrossAttnDownBlockSpatioTemporal", "DownBlockSpatioTemporal"),
up_block_types=("UpBlockSpatioTemporal", "CrossAttnUpBlockSpatioTemporal"),
cross_attention_dim=(32,), num_attention_heads=8,
projection_class_embeddings_input_dim=96, addition_time_embed_dim=32,
)
except Exception as e:
print(type(e).__name__, str(e))
# IndexError tuple index out of range
```

Relevant precedent:
The same initializer already validates `num_attention_heads` and `layers_per_block` sequence lengths before indexing:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/unets/unet_spatio_temporal_condition.py#L112-L124

Suggested fix:
```python
if not isinstance(cross_attention_dim, int) and len(cross_attention_dim) != len(down_block_types):
raise ValueError(
f"Must provide the same number of `cross_attention_dim` as `down_block_types`. "
f"`cross_attention_dim`: {cross_attention_dim}. `down_block_types`: {down_block_types}."
)

if not isinstance(transformer_layers_per_block, int) and len(transformer_layers_per_block) != len(down_block_types):
raise ValueError(
f"Must provide the same number of `transformer_layers_per_block` as `down_block_types`. "
f"`transformer_layers_per_block`: {transformer_layers_per_block}. `down_block_types`: {down_block_types}."
)
```

Start with src/diffusers/pipelines/stable_video_diffusion/pipeline_stable_video_diffusion.py and its return, guidance, image, and latent entry points, then inspect the related UNet initializer in src/diffusers/models/unets/unet_spatio_temporal_condition.py. Run tests/pipelines/stable_video_diffusion/test_stable_video_diffusion.py and add regression coverage for the reported pipeline cases and invalid tuple configurations. Done means the five reproductions follow the documented contracts without runtime or indexing errors.

Written by the indexing model from the issue text.