huggingface / huggingface/diffusers

sana_video model/pipeline review

Open
#13,606 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

# `sana_video` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules. Note: `.ai/review-rules.md` references `AGENTS.md`, but no `AGENTS.md` exists in this checkout; the other referenced rule files were applied.

Duplicate-search status: searched GitHub Issues/PRs for `sana_video`, `SanaVideoPipeline`, `SanaImageToVideoPipeline`, `SanaVideoTransformer3DModel`, `num_videos_per_prompt`, `conditioning_mask`, `image_latents`, `attention_mask`, `cross_attention_dim`, and the docs URL typo. I found related merged PRs `#12584`, `#12634`, `#13229`, `#12675`, and issue `#12760`, but no exact duplicate for the findings below.

Test coverage status: fast model/pipeline tests exist, but slow Sana Video tests are skipped TODOs; see Issue 5. Direct repro snippets were run with `.venv`. Pytest collection is currently blocked in this `.venv` by the installed torch build missing `torch._C._distributed_c10d`.

## Issue 1: I2V fails for `num_videos_per_prompt > 1`

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/sana_video/pipeline_sana_video_i2v.py#L953-L975
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/sana_video/pipeline_sana_video_i2v.py#L990-L1004

Problem:
`latents` are prepared for `batch_size * num_videos_per_prompt`, but `conditioning_mask` is created with only `batch_size`. The timestep tensor is therefore too small for the transformer batch and reshapes to the wrong token length.

Impact:
`SanaImageToVideoPipeline` cannot generate multiple videos per prompt.

Reproduction:
```python
import torch
from PIL import Image
from diffusers import AutoencoderKLWan, FlowMatchEulerDiscreteScheduler, SanaImageToVideoPipeline, SanaVideoTransformer3DModel

vae = AutoencoderKLWan(base_dim=3, z_dim=16, dim_mult=[1, 1, 1, 1], num_res_blocks=1, temperal_downsample=[False, True, True])
transformer = SanaVideoTransformer3DModel(in_channels=16, out_channels=16, num_attention_heads=2, attention_head_dim=12, num_layers=1, num_cross_attention_heads=2, cross_attention_head_dim=12, cross_attention_dim=24, caption_channels=8, sample_size=8, patch_size=(1, 2, 2), rope_max_seq_len=32)
pipe = SanaImageToVideoPipeline(None, None, vae, transformer, FlowMatchEulerDiscreteScheduler())
pipe.set_progress_bar_config(disable=True)
pipe(
image=Image.new("RGB", (32, 32)),
prompt_embeds=torch.randn(1, 16, 8),
prompt_attention_mask=torch.ones(1, 16, dtype=torch.long),
height=32, width=32, frames=9, num_inference_steps=1,
guidance_scale=1.0, num_videos_per_prompt=2,
output_type="latent", use_resolution_binning=False,
)
```

Relevant precedent:
`SanaVideoPipeline` expands timestep from `latent_model_input.shape[0]`:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/sana_video/pipeline_sana_video.py#L944-L947

Suggested fix:
```python
effective_batch_size = batch_size * num_videos_per_prompt
conditioning_mask = latents.new_zeros(
effective_batch_size,
1,
latents.shape[2] // self.transformer_temporal_patch_size,
latents.shape[3] // self.transformer_spatial_patch_size,
latents.shape[4] // self.transformer_spatial_patch_size,
)
conditioning_mask[:, :, 0] = 1.0
if self.do_classifier_free_guidance:
conditioning_mask = torch.cat([conditioning_mask, conditioning_mask])
```

## Issue 2: I2V batched images are broken

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/sana_video/pipeline_sana_video_i2v.py#L465-L466
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/sana_video/pipeline_sana_video_i2v.py#L694-L715

Problem:
The signature accepts `PipelineImageInput`, but validation rejects image lists. Batched tensor images pass validation, then `prepare_latents` repeats encoded image latents by the effective batch size, producing too many image latents.

Impact:
Batched I2V inference fails for normal batched tensor inputs and cannot accept list-style image batches.

Reproduction:
```python
import torch
from diffusers import AutoencoderKLWan, SanaImageToVideoPipeline

pipe = SanaImageToVideoPipeline.__new__(SanaImageToVideoPipeline)
pipe.vae = AutoencoderKLWan(base_dim=3, z_dim=16, dim_mult=[1, 1, 1, 1], num_res_blocks=1, temperal_downsample=[False, True, True])
pipe.vae_scale_factor_temporal = pipe.vae.config.scale_factor_temporal
pipe.vae_scale_factor_spatial = pipe.vae.config.scale_factor_spatial

pipe.prepare_latents(
image=torch.zeros(2, 3, 32, 32),
batch_size=2,
num_channels_latents=16,
height=32,
width=32,
num_frames=9,
dtype=torch.float32,
device=torch.device("cpu"),
)
```

Relevant precedent:
CogVideoX I2V accepts list inputs in validation:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox_image2video.py#L467-L474

Suggested fix:
```python
image_latents = retrieve_latents(self.vae.encode(image), sample_mode="argmax")
if image_latents.shape[0] != batch_size:
if batch_size % image_latents.shape[0] != 0:
raise ValueError("Image batch size must divide the effective prompt batch size.")
image_latents = image_latents.repeat_interleave(batch_size // image_latents.shape[0], dim=0)
```
Also update validation to accept valid `PipelineImageInput` lists/arrays.

## Issue 3: `attention_mask` is accepted but ignored by self-attention

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_sana_video.py#L596-L607
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_sana_video.py#L638-L668
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_sana_video.py#L433-L441

Problem:
The model converts `attention_mask` and passes it into each block, but `SanaVideoTransformerBlock` never forwards it to `attn1`; only `encoder_attention_mask` is used for cross-attention.

Impact:
Callers can pass latent-token masks and get silently unmasked output.

Reproduction:
```python
import torch
from diffusers import SanaVideoTransformer3DModel

torch.manual_seed(0)
model = SanaVideoTransformer3DModel(in_channels=16, out_channels=16, num_attention_heads=2, attention_head_dim=12, num_layers=1, num_cross_attention_heads=2, cross_attention_head_dim=12, cross_attention_dim=24, caption_channels=8, sample_size=8, patch_size=(1, 2, 2), rope_max_seq_len=32).eval()
hidden_states = torch.randn(1, 16, 2, 8, 8)
encoder_hidden_states = torch.randn(1, 12, 8)
timestep = torch.tensor([1])
mask = torch.zeros(1, 32, dtype=torch.long)

with torch.no_grad():
a = model(hidden_states, encoder_hidden_states, timestep, return_dict=False)[0]
b = model(hidden_states, encoder_hidden_states, timestep, attention_mask=mask, return_dict=False)[0]
print((a - b).abs().max().item()) # 0.0
```

Relevant precedent:
The model review rules say declared masks must be honored or omitted.

Suggested fix:
Route `attention_mask` into `attn1` and implement padding-mask support in `SanaLinearAttnProcessor3_0`, preferably keeping boolean masks until the processor. If self-attention masks are not supported, remove `attention_mask` from the public forward signatures.

## Issue 4: `cross_attention_dim=None` is accepted but crashes

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_sana_video.py#L510-L512
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_sana_video.py#L389-L446

Problem:
`cross_attention_dim` is typed as optional, but when it is `None`, the block never defines `attn2` or `norm2`. Forward then accesses both unconditionally.

Impact:
A serialized config with `cross_attention_dim: null` loads but cannot run.

Reproduction:
```python
import torch
from diffusers import SanaVideoTransformer3DModel

model = SanaVideoTransformer3DModel(in_channels=16, out_channels=16, num_attention_heads=2, attention_head_dim=12, num_layers=1, cross_attention_dim=None, caption_channels=8, sample_size=8, patch_size=(1, 2, 2), rope_max_seq_len=32)
model(
hidden_states=torch.randn(1, 16, 2, 8, 8),
encoder_hidden_states=torch.randn(1, 12, 8),
timestep=torch.tensor([1]),
return_dict=False,
)
```

Relevant precedent:
Other transformer blocks either make cross-attention required or initialize optional attention attributes to `None`.

Suggested fix:
```python
self.norm2 = nn.LayerNorm(dim, elementwise_affine=norm_elementwise_affine, eps=norm_eps)
self.attn2 = None
if cross_attention_dim is not None:
self.attn2 = Attention(...)
```

## Issue 5: Slow tests are skipped TODOs

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/sana_video/test_sana_video.py#L208-L225
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/sana_video/test_sana_video_i2v.py#L221-L238

Problem:
Both slow integration tests are present only as `@unittest.skip("TODO: test needs to be implemented")`.

Impact:
There is no real slow coverage for published Sana Video checkpoints, scheduler/config loading, or end-to-end output stability.

Reproduction:
```python
from pathlib import Path

for path in [
"tests/pipelines/sana_video/test_sana_video.py",
"tests/pipelines/sana_video/test_sana_video_i2v.py",
]:
text = Path(path).read_text()
print(path, '@unittest.skip("TODO: test needs to be implemented")' in text)
```

Relevant precedent:
Wan video pipelines include slow tests with expected output slices, e.g. `tests/pipelines/wan/test_wan.py`.

Suggested fix:
Implement T2V and I2V slow tests against the published `Efficient-Large-Model/SANA-Video_2B_480p_diffusers` checkpoint with fixed seeds, low step counts, and expected tensor/video slices.

## Issue 6: Docs model link points to a non-existent repo

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/docs/source/en/api/pipelines/sana_video.md#L34

Problem:
The link target is `Efficient-Large-Model/ANA-Video_2B_480p_diffusers`, missing the leading `S`.

Impact:
Users following the docs model table hit the wrong Hugging Face URL.

Reproduction:
```python
from pathlib import Path

text = Path("docs/source/en/api/pipelines/sana_video.md").read_text()
assert "https://huggingface.co/Efficient-Large-Model/ANA-Video_2B_480p_diffusers" in text
```

Relevant precedent:
The examples in the pipeline code use `Efficient-Large-Model/SANA-Video_2B_480p_diffusers`.

Suggested fix:
```md
| [`Efficient-Large-Model/SANA-Video_2B_480p_diffusers`](https://huggingface.co/Efficient-Large-Model/SANA-Video_2B_480p_diffusers) | `torch.bfloat16` |
```

Contributor guide

Open the contributing guide

Research direction

Start with the affected Sana Video files: src/diffusers/pipelines/sana_video/pipeline_sana_video_i2v.py, src/diffusers/models/transformers/transformer_sana_video.py, and the Sana Video tests and docs named in the issue. Run the provided reproductions first; completion means the reported I2V, masking, configuration, test-coverage, and documentation failures are addressed and the relevant tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
documentation, machine-learning, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.