ltx model/pipeline review

ltx model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules. Reviewed the listed LTX pipelines, modular pipeline blocks, LTX VAE, LTX transformer, imports/lazy-loading, tests, docs references, and duplicate GitHub Issues/PRs.

Duplicate search: checked broad ltx plus the specific class names and failure modes. Related but not exact duplicates: #10565 is a broad LTX I2V quality issue, #11104 is an older closed offload issue for LTXPipeline, #13121 fixed a related LTX2 num_videos_per_prompt bug, #13254 only refactors transformer tests, and #13378 introduced the modular LTX pipeline.

Issue 1: T5 prompt encoding builds a padding mask but does not pass it to T5

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/pipeline_ltx.py#L269
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/pipeline_ltx_image2video.py#L292
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/pipeline_ltx_i2v_long_multi_prompt.py#L538
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/modular_pipelines/ltx/encoders.py#L52

Problem:
These paths create prompt_attention_mask but call T5 as text_encoder(input_ids) instead of passing attention_mask=prompt_attention_mask. Padding tokens therefore participate in prompt encoding. LTXConditionPipeline already uses the mask, so prompt encoding is inconsistent within the LTX family.

Impact:
Short prompts padded to max_sequence_length get different embeddings from the masked T5 encoding. This can affect text adherence and may be related to broad I2V quality reports such as #10565, though I did not find an exact duplicate for this failure mode.

Reproduction:

import torch
from transformers import AutoTokenizer, T5EncoderModel
from diffusers import LTXPipeline

model_id = "hf-internal-testing/tiny-random-t5"
tokenizer = AutoTokenizer.from_pretrained(model_id)
text_encoder = T5EncoderModel.from_pretrained(model_id).eval()
pipe = LTXPipeline(None, None, text_encoder, tokenizer, None)

with torch.no_grad():
    got, mask = pipe._get_t5_prompt_embeds(["short"], max_sequence_length=16, device=torch.device("cpu"))
    inputs = tokenizer(["short"], padding="max_length", max_length=16, truncation=True, return_tensors="pt")
    expected = text_encoder(inputs.input_ids, attention_mask=inputs.attention_mask.bool())[0]

print(mask.tolist())
print((got - expected).abs().max().item())

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/pipeline_ltx_condition.py#L355
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/mochi/pipeline_mochi.py#L240

Suggested fix:

prompt_embeds = self.text_encoder(text_input_ids.to(device), attention_mask=prompt_attention_mask)[0]
# modular:
prompt_embeds = components.text_encoder(text_input_ids.to(device), attention_mask=prompt_attention_mask)[0]

Issue 2: use_framewise_encoding never enables temporal tiled encoding

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_ltx.py#L1220-L1224

Problem:
AutoencoderKLLTXVideo._encode checks self.use_framewise_decoding instead of self.use_framewise_encoding. Setting use_framewise_encoding = True does nothing, while setting use_framewise_decoding = True also changes encode behavior.

Impact:
Users cannot enable framewise encoding to reduce VAE encode memory for long videos, and enabling framewise decoding unexpectedly changes encode behavior too.

Reproduction:

import torch
from unittest import mock
from diffusers import AutoencoderKLLTXVideo

vae = AutoencoderKLLTXVideo(
    in_channels=3, out_channels=3, latent_channels=4,
    block_out_channels=(8, 8, 8, 8), decoder_block_out_channels=(8, 8, 8, 8),
    layers_per_block=(1, 1, 1, 1, 1), decoder_layers_per_block=(1, 1, 1, 1, 1),
    spatio_temporal_scaling=(True, True, False, False),
    decoder_spatio_temporal_scaling=(True, True, False, False),
    decoder_inject_noise=(False, False, False, False, False),
    upsample_residual=(False, False, False, False), upsample_factor=(1, 1, 1, 1),
    patch_size=1, patch_size_t=1, encoder_causal=True, decoder_causal=False,
)
vae.tile_sample_min_num_frames = 1
x = torch.randn(1, 3, 3, 32, 32)

vae.use_framewise_encoding = True
vae.use_framewise_decoding = False
with mock.patch.object(vae, "_temporal_tiled_encode", side_effect=RuntimeError("called")):
    vae.encode(x)
    print("framewise encoding did not call temporal tiled encode")

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_hunyuan_video.py#L767-L811

Suggested fix:

if self.use_framewise_encoding and num_frames > self.tile_sample_min_num_frames:
    return self._temporal_tiled_encode(x)

Issue 3: LTXImageToVideoPipeline crashes for generator lists with multiple videos per prompt

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/pipeline_ltx_image2video.py#L534-L548

Problem:
When num_videos_per_prompt > 1, the effective batch size is larger than the image batch. The non-list generator path broadcasts one encoded image, but the list-generator path indexes image[i] for every effective batch item and crashes.

Impact:
Users cannot generate multiple deterministic I2V samples per prompt with a list of generators.

Reproduction:

import torch
from types import SimpleNamespace
from diffusers import LTXImageToVideoPipeline

pipe = object.__new__(LTXImageToVideoPipeline)
pipe.vae_spatial_compression_ratio = 1
pipe.vae_temporal_compression_ratio = 1
pipe.transformer_spatial_patch_size = 1
pipe.transformer_temporal_patch_size = 1
pipe.vae = SimpleNamespace(
    latents_mean=torch.zeros(8),
    latents_std=torch.ones(8),
    encode=lambda x: SimpleNamespace(latents=torch.zeros(1, 8, 1, 4, 4)),
)

pipe.prepare_latents(
    image=torch.zeros(1, 3, 4, 4),
    batch_size=2,
    num_channels_latents=8,
    height=4,
    width=4,
    num_frames=2,
    dtype=torch.float32,
    device=torch.device("cpu"),
    generator=[torch.Generator().manual_seed(0), torch.Generator().manual_seed(1)],
)

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/modular_pipelines/ltx/before_denoise.py#L366-L369
Related LTX2 coverage precedent: https://github.com/huggingface/diffusers/pull/13121

Suggested fix:
Repeat encoded image latents to the effective batch before sampling per-generator, or index source images modulo the image batch:

source_index = i % image.shape[0]
retrieve_latents(self.vae.encode(image[source_index].unsqueeze(0).unsqueeze(2)), generator[i])

Issue 4: Legacy multi-condition arguments silently drop conditions

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/pipeline_ltx_condition.py#L1029-L1044
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/pipeline_ltx_condition.py#L1074-L1076

Problem:
If image is a list and video is None, the code later rewrites video to [None] and resets num_conditions = 1. The subsequent zip(image, video, frame_index, strength) processes only the first image. The same issue happens for video lists when image is None.

Impact:
Users passing multiple image/frame_index/strength values through the legacy arguments get only the first condition applied, with no error.

Reproduction:

# Mirrors LTXConditionPipeline.__call__ normalization at lines 1029-1044.
image = ["image0", "image1"]
video = None
frame_index = [0, 8]
strength = [1.0, 1.0]

if not isinstance(image, list):
    image = [image]
    num_conditions = 1
elif isinstance(image, list):
    num_conditions = len(image)
if not isinstance(video, list):
    video = [video]
    num_conditions = 1

print(list(zip(image, video, frame_index, strength)))
# Only [('image0', None, 0, 1.0)] is processed.

Relevant precedent:
The conditions=[LTXVideoCondition(...), ...] path preserves list length:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/pipeline_ltx_condition.py#L1021-L1028

Suggested fix:
Normalize the absent modality to the detected condition count instead of resetting the count:

if image is None:
    image = [None] * len(video)
elif not isinstance(image, list):
    image = [image]

if video is None:
    video = [None] * len(image)
elif not isinstance(video, list):
    video = [video]

num_conditions = max(len(image), len(video))

Issue 5: Latent upsample pipeline has no model CPU offload sequence

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/pipeline_ltx_latent_upsample.py#L45

Problem:
LTXLatentUpsamplePipeline.model_cpu_offload_seq is an empty string even though the pipeline runs both vae and latent_upsampler.

Impact:
enable_model_cpu_offload() cannot use an explicit component order for the upsample pipeline, which is exactly the kind of memory-sensitive path users are likely to offload. The old closed issue #11104 is about a different LTX offload failure, not this specific pipeline sequence gap.

Reproduction:

from diffusers import LTXLatentUpsamplePipeline, LTX2LatentUpsamplePipeline

print(repr(LTXLatentUpsamplePipeline.model_cpu_offload_seq))
print(repr(LTX2LatentUpsamplePipeline.model_cpu_offload_seq))

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx2/pipeline_ltx2_latent_upsample.py#L105

Suggested fix:

model_cpu_offload_seq = "vae->latent_upsampler"

Issue 6: Slow tests are missing, and LTXI2VLongMultiPromptPipeline has no fast test

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/ltx/__init__.py#L28-L43

Problem:
The LTX family has fast tests for the main T2V/I2V/condition/latent-upsample pipelines, models, LoRA, and modular assembly, but I found no @slow LTX tests. I also found no test file for LTXI2VLongMultiPromptPipeline.

Impact:
Real checkpoint loading, docs examples, offload behavior, and the long multi-prompt windowing path are not covered. Open PR #13254 touches only transformer tests and does not cover this gap.

Reproduction:

from pathlib import Path

ltx_tests = list(Path("tests").rglob("*ltx*.py"))
print("slow hits:", [str(p) for p in ltx_tests if "@slow" in p.read_text(encoding="utf-8")])
print("long pipeline test exists:", Path("tests/pipelines/ltx/test_ltx_i2v_long_multi_prompt.py").exists())

Relevant precedent:
Pipeline families usually carry at least one slow test for real checkpoint loading/inference when a public pipeline is documented.

Suggested fix:
Add a slow test for a published LTX checkpoint, and add a focused fast test for LTXI2VLongMultiPromptPipeline using tiny local components and output_type="latent".

Issue 7: Modular LTX fast test uses a contributor-owned model repo

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/modular_pipelines/ltx/test_modular_pipeline_ltx.py#L45-L49

Problem:
The modular review rules require tiny test models under hf-internal-testing/, but the LTX modular test uses akshan-main/tiny-ltx-modular-pipe.

Impact:
CI and contributors depend on a personal namespace for a required fast test fixture. PR #13378 introduced the modular pipeline, but I found no follow-up issue/PR moving this fixture.

Reproduction:

from pathlib import Path

text = Path("tests/modular_pipelines/ltx/test_modular_pipeline_ltx.py").read_text()
print("akshan-main/tiny-ltx-modular-pipe" in text)

Relevant precedent:
The local modular rule modular.md explicitly says tiny test models must live under hf-internal-testing/.

Suggested fix:
Move/copy the tiny modular fixture to hf-internal-testing/tiny-ltx-modular-pipe and update the test constant.

Issue 8: Modular generated docstrings still contain TODO placeholders

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/modular_pipelines/ltx/modular_blocks_ltx.py#L33-L66

Problem:
modular_blocks_ltx.py contains many generated TODO: Add description. entries. The modular review rules require running utils/modular_auto_docstring.py --fix_and_overwrite and verifying no TODO placeholders remain.

Impact:
The modular pipeline public docs are incomplete and the block IO contract is less usable for users composing blocks directly.

Reproduction:

from pathlib import Path

for i, line in enumerate(Path("src/diffusers/modular_pipelines/ltx/modular_blocks_ltx.py").read_text().splitlines(), 1):
    if "TODO: Add description" in line:
        print(i, line.strip())

Relevant precedent:
The local modular rules require accurate InputParam/OutputParam descriptions and generated docstrings without TODO placeholders.

Suggested fix:
Add explicit descriptions for the unresolved block inputs/outputs, then run:

python utils/modular_auto_docstring.py --fix_and_overwrite

Start by choosing one numbered defect and reading its linked entry point, such as the LTX pipeline files, autoencoder_kl_ltx.py, or pipeline_ltx_condition.py. Run the issue's reproduction, then inspect the related tests including tests/pipelines/ltx/test_ltx_i2v_long_multi_prompt.py and tests/modular_pipelines/ltx/test_modular_pipeline_ltx.py. Done means the selected behavior is corrected and covered by an appropriate regression or coverage test.