hunyuan_video model/pipeline review

hunyuan_video model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules.

Files/categories reviewed: public exports and lazy imports, pipeline runtime behavior, config/loading surfaces, dtype/device/offload-sensitive paths, model forward behavior, attention processor surfaces, fast/slow tests, and duplicate status.

Duplicate-search status: searched existing huggingface/diffusers Issues and PRs for hunyuan_video, affected class names, prompt_2, prompt_attention_mask, num_videos_per_prompt, and Framepack transformer failure modes. I did not find direct duplicates for the findings below. Related but not duplicate-adjacent batch issues exist for the older text-to-video pipeline, including #10453 and #10542.

Issue 1: prompt_2 is ignored by the CLIP encoder

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video.py#L327-L336
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_skyreels_image2video.py#L356-L365
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video_framepack.py#L433-L442
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video_image2video.py#L483-L492

Problem:
All four pipelines normalize prompt_2 from prompt, but then pass prompt into _get_clip_prompt_embeds(...). As a result, user-provided prompt_2 and negative_prompt_2 never affect the CLIP branch.

Impact:
The public API exposes a second prompt input that silently does nothing. Users cannot independently condition the Llama and CLIP encoders, and negative CLIP prompting is also misapplied.

Reproduction:

import torch
from diffusers import HunyuanVideoPipeline

class Probe(HunyuanVideoPipeline):
    def __init__(self):
        pass

    def _get_llama_prompt_embeds(self, prompt, *args, **kwargs):
        return torch.zeros(1, 1, 1), torch.ones(1, 1)

    def _get_clip_prompt_embeds(self, prompt, *args, **kwargs):
        self.clip_prompt_seen = prompt
        return torch.zeros(1, 1)

pipe = Probe()
pipe.encode_prompt(
    prompt="main prompt",
    prompt_2="clip prompt",
    prompt_template={"template": "{}", "crop_start": 0},
)
print(pipe.clip_prompt_seen)
# main prompt

Relevant precedent:
The same pipelines already intend this split with if prompt_2 is None: prompt_2 = prompt; the bug is the wrong argument at the CLIP callsite.

Suggested fix:

pooled_prompt_embeds = self._get_clip_prompt_embeds(
    prompt_2,
    num_videos_per_prompt,
    device=device,
    dtype=dtype,
    max_sequence_length=77,
)

Apply this in the source pipeline and copied variants, then run the repository copy fixer for copied code.

Issue 2: Precomputed prompt embeds crash without attention masks

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video.py#L340-L386
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video.py#L651-L664
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_skyreels_image2video.py#L370-L416
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video_framepack.py#L455-L505
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video_image2video.py#L496-L550

Problem:
The pipelines allow prompt_embeds to be supplied directly, but check_inputs does not require prompt_attention_mask. encode_prompt returns None for the mask, and __call__ later executes prompt_attention_mask.to(...).

Impact:
The documented embed-based path fails with an opaque AttributeError instead of a validation error. The same gap exists for negative prompt embeddings and pooled embeddings.

Reproduction:

import torch
from diffusers import HunyuanVideoPipeline

pipe = object.__new__(HunyuanVideoPipeline)
prompt_embeds = torch.zeros(1, 2, 3)
pooled_prompt_embeds = torch.zeros(1, 4)

try:
    HunyuanVideoPipeline.check_inputs(
        pipe,
        prompt=None,
        prompt_2=None,
        height=16,
        width=16,
        prompt_embeds=prompt_embeds,
        callback_on_step_end_tensor_inputs=None,
        prompt_template={"template": "{}"},
    )
    _, _, prompt_attention_mask = HunyuanVideoPipeline.encode_prompt(
        pipe,
        prompt=None,
        prompt_embeds=prompt_embeds,
        pooled_prompt_embeds=pooled_prompt_embeds,
        prompt_attention_mask=None,
    )
    prompt_attention_mask.to(torch.float32)
except Exception as e:
    print(type(e).__name__, e)
# AttributeError 'NoneType' object has no attribute 'to'

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/pipeline_ltx.py#L399-L403

Suggested fix:

if prompt_embeds is not None and prompt_attention_mask is None:
    raise ValueError("Must provide `prompt_attention_mask` when specifying `prompt_embeds`.")
if prompt_embeds is not None and pooled_prompt_embeds is None:
    raise ValueError("Must provide `pooled_prompt_embeds` when specifying `prompt_embeds`.")
if negative_prompt_embeds is not None and negative_prompt_attention_mask is None:
    raise ValueError("Must provide `negative_prompt_attention_mask` when specifying `negative_prompt_embeds`.")
if negative_prompt_embeds is not None and negative_pooled_prompt_embeds is None:
    raise ValueError("Must provide `negative_pooled_prompt_embeds` when specifying `negative_prompt_embeds`.")

Issue 3: Image-conditioned pipelines break num_videos_per_prompt > 1

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video_image2video.py#L552-L599
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_skyreels_image2video.py#L417-L459
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video_framepack.py#L551-L564
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video_framepack.py#L908-L961

Problem:
The image-conditioned pipelines compute an effective batch size using batch_size * num_videos_per_prompt, but image latents are encoded once and not expanded to the effective batch. The I2V pipeline also does not repeat prompt embeddings in its prompt encoder helpers. Framepack duplicates text embeddings but still prepares guidance, history latents, and denoising latents using the original batch size.

Impact:
num_videos_per_prompt > 1 can produce tensor batch mismatches or internally inconsistent conditioning, especially when a single image should generate multiple videos.

Reproduction:

import torch
from diffusers import HunyuanVideoImageToVideoPipeline

class Dist:
    def __init__(self, latents):
        self._latents = latents
    def mode(self):
        return self._latents

class EncOut:
    def __init__(self, latents):
        self.latent_dist = Dist(latents)

class VAE:
    def encode(self, x):
        return EncOut(torch.zeros(x.shape[0], 4, 1, 2, 2))

pipe = object.__new__(HunyuanVideoImageToVideoPipeline)
pipe.vae = VAE()
pipe.vae_scaling_factor = 1.0
pipe.vae_scale_factor_temporal = 4
pipe.vae_scale_factor_spatial = 8

latents, image_latents = HunyuanVideoImageToVideoPipeline.prepare_latents(
    pipe,
    image=torch.zeros(1, 3, 16, 16),
    batch_size=2,
    num_channels_latents=4,
    height=16,
    width=16,
    num_frames=9,
    dtype=torch.float32,
    device=torch.device("cpu"),
    generator=None,
    latents=None,
    image_condition_type="latent_concat",
)

print(latents.shape[0], image_latents.shape[0])
# 2 1

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video.py#L256-L261
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video.py#L694-L704

Suggested fix:

def _repeat_to_effective_batch(tensor, effective_batch_size):
    if tensor.shape[0] == effective_batch_size:
        return tensor
    if effective_batch_size % tensor.shape[0] != 0:
        raise ValueError("Conditioning batch size must divide the effective generation batch size.")
    return tensor.repeat_interleave(effective_batch_size // tensor.shape[0], dim=0)

Use this for image latents and image embeddings. Also repeat I2V prompt embeddings like the text-to-video pipeline, and make Framepack use effective_batch_size = batch_size * num_videos_per_prompt consistently for guidance, history, and latent preparation.

Issue 4: Framepack transformer optional config paths crash

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_hunyuan_video_framepack.py#L124-L166
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_hunyuan_video_framepack.py#L236-L269

Problem:
HunyuanVideoFramepackTransformer3DModel defaults has_image_proj=False and has_clean_x_embedder=False, but forward unconditionally calls self.clean_x_embedder(...) and self.image_projection(...). It also passes optional rotary embedding locals that are only defined inside conditional branches.

Impact:
Several serialized config combinations that the class advertises as valid fail at runtime. This weakens config backwards compatibility and makes minimal/tiny test fixtures harder to construct.

Reproduction:

import torch
from diffusers import HunyuanVideoFramepackTransformer3DModel

model = HunyuanVideoFramepackTransformer3DModel(
    in_channels=4,
    out_channels=4,
    num_attention_heads=2,
    attention_head_dim=4,
    num_layers=1,
    num_single_layers=1,
    num_refiner_layers=1,
    patch_size=2,
    patch_size_t=1,
    guidance_embeds=True,
    text_embed_dim=8,
    pooled_projection_dim=6,
    rope_axes_dim=(2, 2, 4),
    has_image_proj=True,
    image_proj_dim=8,
    has_clean_x_embedder=False,
).eval()

try:
    with torch.no_grad():
        model(
            hidden_states=torch.randn(1, 4, 1, 4, 4),
            timestep=torch.tensor([1]),
            encoder_hidden_states=torch.randn(1, 3, 8),
            encoder_attention_mask=torch.ones(1, 3),
            pooled_projections=torch.randn(1, 6),
            image_embeds=torch.randn(1, 2, 8),
            indices_latents=torch.arange(1),
            guidance=torch.tensor([1.0]),
        )
except Exception as e:
    print(type(e).__name__, e)
# TypeError 'NoneType' object is not callable

Relevant precedent:
The non-Framepack Hunyuan transformer keeps optional projections/configured modules aligned with its forward path rather than exposing defaults that immediately fail.

Suggested fix:

image_rotary_emb_clean = None
image_rotary_emb_history_2x = None
image_rotary_emb_history_4x = None

if self.clean_x_embedder is not None:
    latents_clean, latents_history_2x, latents_history_4x = self.clean_x_embedder(
        latents_clean, latents_history_2x, latents_history_4x
    )

if self.image_projection is not None and image_embeds is not None:
    encoder_hidden_states_image = self.image_projection(image_embeds)
    attention_mask_image = encoder_attention_mask.new_ones(
        (batch_size, encoder_hidden_states_image.shape[1])
    )
    encoder_hidden_states = torch.cat([encoder_hidden_states_image, encoder_hidden_states], dim=1)
    encoder_attention_mask = torch.cat([attention_mask_image, encoder_attention_mask], dim=1)

If released Framepack checkpoints always require these modules, the safer alternative is to change the config defaults and validate required inputs early with clear errors.

Issue 5: Slow tests are missing for the Hunyuan Video family

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/hunyuan_video/test_hunyuan_video.py#L348-L357
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/hunyuan_video/test_hunyuan_image2video.py#L374-L389
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/hunyuan_video/test_hunyuan_skyreels_image2video.py#L335-L344
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/hunyuan_video/test_hunyuan_video_framepack.py#L394-L403

Problem:
Fast tests exist for the Hunyuan Video pipelines and models, but I found no @slow pipeline tests under tests/pipelines/hunyuan_video. Several batch consistency tests are explicitly skipped, including for the image-conditioned and Framepack pipelines.

Impact:
Real-checkpoint smoke coverage is missing for this family, and the skipped batch tests leave the num_videos_per_prompt regressions above unguarded.

Reproduction:

from pathlib import Path

files = sorted(Path("tests/pipelines/hunyuan_video").glob("test_*.py"))
print({str(path): ("@slow" in path.read_text()) for path in files})
# All entries are False.

Relevant precedent:
Other major pipeline families include slow smoke tests for real checkpoint loading, basic inference, and scheduler/device behavior.

Suggested fix:
Add slow smoke tests for HunyuanVideoPipeline, HunyuanVideoImageToVideoPipeline, HunyuanSkyreelsImageToVideoPipeline, and HunyuanVideoFramepackPipeline. Add focused fast coverage for prompt_2, precomputed prompt embeds with masks, and num_videos_per_prompt=2; then unskip the batch consistency tests once the pipeline behavior is fixed.

Start with the four affected Hunyuan Video pipeline files and transformer_hunyuan_video_framepack.py, then run the reproductions for prompt_2, precomputed embeddings, repeated videos, and optional Framepack configuration. Compare the referenced non-Framepack pipeline and LTX validation behavior. Done means the four reported paths work across the listed variants, with copied-code synchronization and regression coverage for each failure.