animatediff model/pipeline review

animatediff model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules. Public exports/imports, config/load paths, dtype/device/offload behavior, model attention behavior, docs/examples, and fast/slow tests were checked. Reproductions were run with .venv.

Duplicate search: checked GitHub Issues and PRs for animatediff, affected classes/files, num_videos_per_prompt, MultiControlNet validation, stale unet_motion_model import, SparseControlNet tests, and slow-test coverage. No exact duplicates found. Related but not duplicates: #8664, #9326, #9508, #7378.

Issue 1: Video-to-video pipelines ignore num_videos_per_prompt

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/animatediff/pipeline_animatediff_video2video.py#L752-L864
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/animatediff/pipeline_animatediff_video2video_controlnet.py#L918-L1062

Problem:
Both public signatures expose num_videos_per_prompt, but __call__ overwrites it with 1 before input validation, latent preparation, prompt expansion, and denoising. Users requesting multiple videos per prompt silently receive one video.

Impact:
Batch semantics are wrong and no test catches it. This also hides related latent-preparation gaps that need to duplicate/expand the input video latents for num_videos_per_prompt > 1.

Reproduction:

# Run from repo root with: .venv/Scripts/python.exe
# Shows actual batch is 1 even though num_videos_per_prompt=2.
# Uses the same tiny component shapes as the fast tests.
import torch
from PIL import Image
from transformers import CLIPTextConfig, CLIPTextModel, CLIPTokenizer
from diffusers import AutoencoderKL, DDIMScheduler, MotionAdapter, UNet2DConditionModel, AnimateDiffVideoToVideoPipeline

dim, blocks = 8, (8, 8)
unet = UNet2DConditionModel(block_out_channels=blocks, layers_per_block=2, sample_size=8, in_channels=4, out_channels=4, down_block_types=("CrossAttnDownBlock2D", "DownBlock2D"), up_block_types=("CrossAttnUpBlock2D", "UpBlock2D"), cross_attention_dim=dim, norm_num_groups=2)
vae = AutoencoderKL(block_out_channels=blocks, in_channels=3, out_channels=3, down_block_types=["DownEncoderBlock2D"] * 2, up_block_types=["UpDecoderBlock2D"] * 2, latent_channels=4, norm_num_groups=2)
text_encoder = CLIPTextModel(CLIPTextConfig(bos_token_id=0, eos_token_id=2, hidden_size=dim, intermediate_size=37, num_attention_heads=4, num_hidden_layers=5, pad_token_id=1, vocab_size=1000))
pipe = AnimateDiffVideoToVideoPipeline(unet=unet, scheduler=DDIMScheduler(), vae=vae, motion_adapter=MotionAdapter(block_out_channels=blocks, motion_layers_per_block=2, motion_norm_num_groups=2, motion_num_attention_heads=4), text_encoder=text_encoder, tokenizer=CLIPTokenizer.from_pretrained("hf-internal-testing/tiny-random-clip"), feature_extractor=None, image_encoder=None)
pipe.set_progress_bar_config(disable=True)
out = pipe(video=[Image.new("RGB", (32, 32)) for _ in range(2)], prompt="test", num_inference_steps=1, strength=1.0, num_videos_per_prompt=2, output_type="pt").frames
print(out.shape[0])  # expected 2, actual 1

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py#L735-L783

Suggested fix:
Remove the hard reset in both pipelines and update prepare_latents to expand the input video batch like img2img does:

# remove this line from both __call__ methods
num_videos_per_prompt = 1

# in prepare_latents, before encoding with a generator list
if isinstance(generator, list) and video.shape[0] < batch_size and batch_size % video.shape[0] == 0:
    video = torch.cat([video] * (batch_size // video.shape[0]), dim=0)

# after init_latents is built
if batch_size > init_latents.shape[0] and batch_size % init_latents.shape[0] == 0:
    init_latents = torch.cat([init_latents] * (batch_size // init_latents.shape[0]), dim=0)

Issue 2: AnimateDiff Multi-ControlNet validation can silently drop ControlNets

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/animatediff/pipeline_animatediff_controlnet.py#L562-L592
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/animatediff/pipeline_animatediff_video2video_controlnet.py#L686-L719

Problem:
For MultiControlNetModel, the conditioning-frame list length is not checked against len(controlnet.nets). The scale length check is also unreachable because it is under elif isinstance(controlnet_conditioning_scale, list) after an if isinstance(..., list) branch.

Impact:
A user can pass two ControlNets but only one conditioning video or one scale. Validation succeeds, then MultiControlNetModel.forward zips the lists and silently skips the extra ControlNet.

Reproduction:

import torch
from diffusers import AnimateDiffControlNetPipeline, AnimateDiffVideoToVideoControlNetPipeline, MultiControlNetModel

multi = MultiControlNetModel([torch.nn.Identity(), torch.nn.Identity()])

pipe = object.__new__(AnimateDiffControlNetPipeline)
pipe.controlnet = multi
pipe.check_inputs(prompt="x", height=64, width=64, num_frames=2, video=[[object(), object()]], controlnet_conditioning_scale=[1.0], control_guidance_start=[0.0, 0.0], control_guidance_end=[1.0, 1.0])
print("text2video validation passed unexpectedly")

pipe = object.__new__(AnimateDiffVideoToVideoControlNetPipeline)
pipe.controlnet = multi
pipe.check_inputs(prompt="x", height=64, width=64, strength=0.8, video=[object(), object()], conditioning_frames=[[object(), object()]], latents=None, controlnet_conditioning_scale=[1.0], control_guidance_start=[0.0, 0.0], control_guidance_end=[1.0, 1.0])
print("video2video validation passed unexpectedly")

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/controlnet_sd3/pipeline_stable_diffusion_3_controlnet_inpainting.py#L732-L752
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/controlnets/multicontrolnet.py#L47-L48

Suggested fix:

if isinstance(controlnet, MultiControlNetModel):
    if len(video) != len(controlnet.nets):
        raise ValueError(
            f"For multiple controlnets: expected {len(controlnet.nets)} conditioning videos, got {len(video)}."
        )

    if isinstance(controlnet_conditioning_scale, list):
        if any(isinstance(i, list) for i in controlnet_conditioning_scale):
            raise ValueError("A single batch of multiple conditionings is not supported at the moment.")
        if len(controlnet_conditioning_scale) != len(controlnet.nets):
            raise ValueError(
                "`controlnet_conditioning_scale` must have the same length as the number of controlnets."
            )

Issue 3: Community AnimateDiff image-to-video example imports a removed module path

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/community/pipeline_animatediff_img2video.py#L29-L34

Problem:
The example imports MotionAdapter from diffusers.models.unet_motion_model, but that module path does not exist. The public import is available from diffusers.

Impact:
The community pipeline fails at import time before users can run it.

Reproduction:

from diffusers.models.unet_motion_model import MotionAdapter
# ModuleNotFoundError: No module named 'diffusers.models.unet_motion_model'

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/community/pipeline_animatediff_controlnet.py#L26-L29

Suggested fix:

from diffusers import MotionAdapter

Issue 4: Missing slow coverage for most AnimateDiff variants and missing model tests for SparseControlNetModel

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/animatediff/test_animatediff.py#L560-L562
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/animatediff/test_animatediff_controlnet.py#L42-L45
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/animatediff/test_animatediff_sparsectrl.py#L41-L44
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/animatediff/test_animatediff_sdxl.py#L34-L40
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/controlnets/controlnet_sparsectrl.py#L96-L161

Problem:
Only the base AnimateDiffPipeline has a slow test. There are no slow tests for ControlNet, SparseCtrl, SDXL, video-to-video, or video-to-video ControlNet. SparseControlNetModel also has no model-level test under tests/models/controlnets, so model save/load, config roundtrip, attention processor behavior, and gradient checkpointing are only indirectly covered by pipeline tests.

Impact:
Checkpoint-specific regressions and model serialization issues can ship without coverage. This is especially risky for SparseCtrl because the model is public and loadable independently from the pipeline.

Reproduction:

from pathlib import Path

text = "\n".join(p.read_text(encoding="utf-8") for p in Path("tests").rglob("test*.py"))
for name in [
    "AnimateDiffPipelineSlowTests",
    "AnimateDiffControlNetPipelineSlowTests",
    "AnimateDiffSparseControlNetPipelineSlowTests",
    "AnimateDiffPipelineSDXLSlowTests",
    "AnimateDiffVideoToVideoPipelineSlowTests",
    "AnimateDiffVideoToVideoControlNetPipelineSlowTests",
    "SparseControlNetModelTests",
]:
    print(name, name in text)

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/models/unets/test_models_unet_motion.py#L41-L42
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/animatediff/test_animatediff.py#L560-L621

Suggested fix:
Add slow smoke tests for the missing public pipelines using the documented small checkpoint paths where possible, and add tests/models/controlnets/test_models_controlnet_sparsectrl.py with the standard ModelTesterMixin coverage for forward shape, save/load, variant save/load, attention processors, and gradient checkpointing.

Start with the cited AnimateDiff pipeline files, especially the video-to-video call/prepare_latents and ControlNet check_inputs paths, then run the provided reproductions. Check the community import and the named pipeline/model test files. Done means correcting the reported batch and validation behavior, fixing the import, and adding the requested slow and SparseControlNet coverage.