cogvideo model/pipeline review

Đang mở

#13,622 mở ngày 29/4/2026 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Ngôn ngữ chính: Python
Star: 34.5k
Fork: 7.3k
Merge trung bình: 3 ngày 3 giờ
Pull request đã merge (30 ngày): 91

Mô tả

# `cogvideo` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

Reviewed target pipelines, model files, lazy exports, docs/tests/examples, dtype/device paths, offload-facing behavior, attention processors, and coverage. Public imports/lazy loading looked consistent. I did not find separate actionable issues in `pipeline_output.py` or `autoencoder_kl_cogvideox.py`.

Execution: standalone repros were run with `.venv/Scripts/python.exe`; no full pytest suite was run.

Duplicate search: searched GitHub Issues and PRs for `cogvideo`, affected class/file names, and the specific failure modes. Exact duplicate found only for Issue 5: https://github.com/huggingface/diffusers/issues/9641. Related but not exact: #11133, #9972, #13586, PR #11368, PR #9333.

## Issue 1: `num_videos_per_prompt` is accepted but ignored

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox.py#L518
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox.py#L618
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox_fun_control.py#L564
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox_fun_control.py#L669
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox_image2video.py#L612
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox_image2video.py#L714
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox_video2video.py#L589
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox_video2video.py#L688

Problem:
All four pipelines expose `num_videos_per_prompt`, but each `__call__` resets it to `1` before prompt encoding and latent preparation. The text-only pipeline can already use the parameter correctly if that reset is removed. The conditioned pipelines also need image/video/control latents expanded to the effective batch, or they should reject values above `1`.

Impact:
Users requesting multiple videos per prompt silently get one video per prompt. Batch behavior and callback tensor shapes are also misleading.

Reproduction:
```python
import torch
from diffusers import AutoencoderKLCogVideoX, CogVideoXDDIMScheduler, CogVideoXPipeline, CogVideoXTransformer3DModel

def tiny_pipe():
transformer = CogVideoXTransformer3DModel(
num_attention_heads=4, attention_head_dim=8, in_channels=4, out_channels=4,
time_embed_dim=2, text_embed_dim=32, num_layers=1,
sample_width=2, sample_height=2, sample_frames=9, patch_size=2,
temporal_compression_ratio=4, max_text_seq_length=16,
)
vae = AutoencoderKLCogVideoX(
in_channels=3, out_channels=3,
down_block_types=("CogVideoXDownBlock3D",) * 4,
up_block_types=("CogVideoXUpBlock3D",) * 4,
block_out_channels=(8, 8, 8, 8), latent_channels=4,
layers_per_block=1, norm_num_groups=2, temporal_compression_ratio=4,
)
return CogVideoXPipeline(None, None, transformer, vae, CogVideoXDDIMScheduler())

pipe = tiny_pipe()
frames = pipe(
prompt_embeds=torch.zeros(1, 16, 32),
height=16, width=16, num_frames=5,
num_inference_steps=1, guidance_scale=1,
num_videos_per_prompt=2, output_type="latent",
).frames
print(frames.shape)
assert frames.shape[0] == 2, f"expected 2 videos, got {frames.shape[0]}"
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan.py#L533-L560
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/mochi/pipeline_mochi.py#L635-L650

Suggested fix:
```python
# Text-to-video: remove the forced reset.
# num_videos_per_prompt = 1

# Conditioned pipelines should either expand conditioning latents:
def _repeat_to_effective_batch(tensor, batch_size, num_videos_per_prompt):
if tensor.shape[0] == 1:
return tensor.repeat_interleave(batch_size * num_videos_per_prompt, dim=0)
if tensor.shape[0] == batch_size:
return tensor.repeat_interleave(num_videos_per_prompt, dim=0)
return tensor

# Or reject unsupported requests until expansion is implemented:
if num_videos_per_prompt != 1:
raise ValueError("`num_videos_per_prompt > 1` is not currently supported by this conditioned CogVideoX pipeline.")
```

## Issue 2: `CogVideoXFunControlPipeline` crashes when `control_video_latents` is supplied

Problem:
The API documents `control_video_latents`, and `check_inputs` only rejects passing both raw control video and latents. But `__call__` unconditionally runs `prepare_control_latents(None, control_video)` after the preprocessing branch. When only `control_video_latents` is supplied, `control_video` is `None`, so the user tensor is discarded and the next `.permute()` crashes.

Impact:
The precomputed control-latent path is unusable.

Reproduction:
```python
import torch
from diffusers import AutoencoderKLCogVideoX, CogVideoXDDIMScheduler, CogVideoXFunControlPipeline, CogVideoXTransformer3DModel

transformer = CogVideoXTransformer3DModel(
num_attention_heads=4, attention_head_dim=8, in_channels=8, out_channels=4,
time_embed_dim=2, text_embed_dim=32, num_layers=1,
sample_width=2, sample_height=2, sample_frames=9, patch_size=2,
temporal_compression_ratio=4, max_text_seq_length=16,
)
vae = AutoencoderKLCogVideoX(
in_channels=3, out_channels=3,
down_block_types=("CogVideoXDownBlock3D",) * 4,
up_block_types=("CogVideoXUpBlock3D",) * 4,
block_out_channels=(8, 8, 8, 8), latent_channels=4,
layers_per_block=1, norm_num_groups=2, temporal_compression_ratio=4,
)
pipe = CogVideoXFunControlPipeline(None, None, transformer, vae, CogVideoXDDIMScheduler())

try:
pipe(
prompt_embeds=torch.zeros(1, 16, 32),
control_video_latents=torch.zeros(1, 4, 2, 2, 2),
height=16, width=16, num_inference_steps=1,
guidance_scale=1, output_type="latent",
)
except Exception as e:
print(type(e).__name__, e)
```

Relevant precedent:
The raw-image/video latent paths in the other CogVideoX pipelines keep the precomputed `latents` branch separate from preprocessing.

Suggested fix:
```python
if control_video_latents is None:
if control_video is None:
raise ValueError("Provide either `control_video` or `control_video_latents`.")
control_video = self.video_processor.preprocess_video(control_video, height=height, width=width)
control_video = control_video.to(device=device, dtype=prompt_embeds.dtype)
_, control_video_latents = self.prepare_control_latents(None, control_video)
else:
control_video_latents = control_video_latents.to(device=device, dtype=prompt_embeds.dtype)

control_video_latents = control_video_latents.permute(0, 2, 1, 3, 4)
```

## Issue 3: Supplied `prompt_embeds` and `latents` are not cast to execution dtype

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox.py#L282-L323
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox.py#L342-L348
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox_fun_control.py#L253-L332
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox_fun_control.py#L352-L358
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox_image2video.py#L263-L342
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox_image2video.py#L410-L416
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox_video2video.py#L269-L348
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogvideo/pipeline_cogvideox_video2video.py#L390-L396

Problem:
Generated prompt embeddings are cast, but user-supplied `prompt_embeds` and `negative_prompt_embeds` are returned unchanged. User-supplied `latents` are moved to device but not dtype. With a bf16/fp16 pipeline and fp32 tensors, transformer projections hit dtype mismatches.

Impact:
Documented advanced inputs break mixed precision inference.

Reproduction:
```python
import torch
from diffusers import AutoencoderKLCogVideoX, CogVideoXDDIMScheduler, CogVideoXPipeline, CogVideoXTransformer3DModel

transformer = CogVideoXTransformer3DModel(
num_attention_heads=4, attention_head_dim=8, in_channels=4, out_channels=4,
time_embed_dim=2, text_embed_dim=32, num_layers=1,
sample_width=2, sample_height=2, sample_frames=9, patch_size=2,
temporal_compression_ratio=4, max_text_seq_length=16,
)
vae = AutoencoderKLCogVideoX(
in_channels=3, out_channels=3,
down_block_types=("CogVideoXDownBlock3D",) * 4,
up_block_types=("CogVideoXUpBlock3D",) * 4,
block_out_channels=(8, 8, 8, 8), latent_channels=4,
layers_per_block=1, norm_num_groups=2, temporal_compression_ratio=4,
)
pipe = CogVideoXPipeline(None, None, transformer, vae, CogVideoXDDIMScheduler()).to(dtype=torch.bfloat16)

try:
pipe(
prompt_embeds=torch.zeros(1, 16, 32, dtype=torch.float32),
height=16, width=16, num_frames=5,
num_inference_steps=1, guidance_scale=1, output_type="latent",
)
except RuntimeError as e:
print(e)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan.py#L544-L547
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/mochi/pipeline_mochi.py#L461-L462

Suggested fix:
```python
# After prompt embedding selection in encode_prompt:
prompt_embeds = prompt_embeds.to(device=device, dtype=dtype)
if negative_prompt_embeds is not None:
negative_prompt_embeds = negative_prompt_embeds.to(device=device, dtype=dtype)

# In prepare_latents branches:
latents = latents.to(device=device, dtype=dtype)
```

## Issue 4: Spatial validation accepts sizes that later fail patchification

Problem:
Pipelines only require `height` and `width` to be divisible by `8`, the VAE scale factor. The transformer then patchifies latents with `patch_size=2`, so the original size must usually be divisible by `8 * 2 = 16`. For example, `24x24` passes validation but produces latent `3x3`, which fails later.

Impact:
Users get a late low-level tensor shape error instead of an actionable validation error.

Reproduction:
```python
import torch
from diffusers import AutoencoderKLCogVideoX, CogVideoXDDIMScheduler, CogVideoXPipeline, CogVideoXTransformer3DModel

try:
pipe(
prompt_embeds=torch.zeros(1, 16, 32),
height=24, width=24, num_frames=5,
num_inference_steps=1, guidance_scale=1, output_type="latent",
)
except Exception as e:
print(type(e).__name__, e)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan.py#L497-L511
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video.py#L351

Suggested fix:
```python
spatial_multiple = self.vae_scale_factor_spatial * self.transformer.config.patch_size
if height % spatial_multiple != 0 or width % spatial_multiple != 0:
raise ValueError(
f"`height` and `width` have to be divisible by {spatial_multiple} "
f"because CogVideoX patchifies VAE latents with patch_size={self.transformer.config.patch_size}; "
f"got {height} and {width}."
)
```

## Issue 5: Attention backend selection cannot affect CogVideoX attention

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/cogvideox_transformer_3d.py#L26
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/cogvideox_transformer_3d.py#L95-L104
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/cogvideox_transformer_3d.py#L334-L354
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/attention_processor.py#L2277-L2330
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/attention_processor.py#L2346-L2401

Problem:
`CogVideoXAttnProcessor2_0` and `FusedCogVideoXAttnProcessor2_0` are shared legacy processors without `_attention_backend` / `_parallel_config`, and they call `F.scaled_dot_product_attention` directly. `CogVideoXTransformer3DModel.set_attention_backend(...)` therefore leaves them unchanged and cannot route through the dispatcher.

Impact:
CogVideoX cannot use the model-level attention backend infrastructure consistently, including alternate kernels and parallel attention integrations covered by the review rules.

Reproduction:
```python
from diffusers import CogVideoXTransformer3DModel

model = CogVideoXTransformer3DModel(
num_attention_heads=2, attention_head_dim=8, in_channels=4, out_channels=4,
time_embed_dim=2, text_embed_dim=8, num_layers=1,
sample_width=8, sample_height=8, sample_frames=8,
patch_size=2, temporal_compression_ratio=4, max_text_seq_length=8,
)
model.set_attention_backend("native")
processors = list(model.attn_processors.values())
print([type(p).__name__ for p in processors])
print([hasattr(p, "_attention_backend") for p in processors])
assert all(hasattr(p, "_attention_backend") for p in processors)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_wan.py#L69-L143
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_flux.py#L29-L123

Suggested fix:
Move CogVideoX attention processors into `cogvideox_transformer_3d.py` as model-local processors, add `_attention_backend` and `_parallel_config`, and replace direct SDPA calls with dispatcher calls while preserving text/video split and RoPE behavior:
```python
hidden_states = dispatch_attention_fn(
query,
key,
value,
attn_mask=attention_mask,
dropout_p=0.0,
is_causal=False,
backend=self._attention_backend,
parallel_config=self._parallel_config,
)
```

## Issue 6: Dynamic CFG uses scheduler timestep values as denoising progress

Problem:
This is already tracked by https://github.com/huggingface/diffusers/issues/9641, so I am not presenting it as new. The dynamic CFG formula uses `t.item()` from scheduler timesteps, not the denoising loop index/progress. With standard timesteps like `980, 960, ...`, the expression does not represent normalized progress and can exceed the requested guidance scale because it computes `1 + guidance_scale * ...`.

Impact:
`use_dynamic_cfg=True` applies an unintuitive and scheduler-dependent guidance schedule across all CogVideoX pipelines.

Reproduction:
```python
import math
from diffusers import CogVideoXDDIMScheduler

guidance_scale = 6
num_inference_steps = 50
scheduler = CogVideoXDDIMScheduler()
scheduler.set_timesteps(num_inference_steps)

scales = [
1 + guidance_scale * (
(1 - math.cos(math.pi * ((num_inference_steps - t.item()) / num_inference_steps) ** 5.0)) / 2
)
for t in scheduler.timesteps
]
print(scheduler.timesteps[:5].tolist())
print([round(x, 3) for x in scales[:12]], max(scales))
assert max(scales) <= guidance_scale
```

Relevant precedent:
The intended behavior should be resolved in the existing issue: https://github.com/huggingface/diffusers/issues/9641

Suggested fix:
Use loop progress instead of scheduler timestep value, and confirm the intended max scale against the CogVideoX implementation:
```python
progress = (num_inference_steps - i) / num_inference_steps
self._guidance_scale = 1 + (guidance_scale - 1) * ((1 - math.cos(math.pi * progress**5.0)) / 2)
```

## Issue 7: Slow tests are missing for FunControl and Video-to-Video

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/cogvideo/test_cogvideox_fun_control.py#L1-L330
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/cogvideo/test_cogvideox_video2video.py#L1-L326

Problem:
Fast tests exist for these pipelines, but there are no `@slow` integration tests for `CogVideoXFunControlPipeline` or `CogVideoXVideoToVideoPipeline`. Text-to-video and image-to-video do have slow coverage.

Impact:
Checkpoint compatibility, preprocessing with real media, and end-to-end output regressions are not covered for two public CogVideoX pipelines.

Reproduction:
```python
from pathlib import Path

for path in [
"tests/pipelines/cogvideo/test_cogvideox_fun_control.py",
"tests/pipelines/cogvideo/test_cogvideox_video2video.py",
]:
text = Path(path).read_text()
print(path, "@slow" in text, "IntegrationTests" in text)
assert "@slow" in text and "IntegrationTests" in text
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/cogvideo/test_cogvideox.py#L339-L357
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/cogvideo/test_cogvideox_image2video.py#L351-L369

Suggested fix:
Add slow integration classes for `CogVideoXVideoToVideoPipeline` and `CogVideoXFunControlPipeline` with published checkpoints, fixed seeds, small media fixtures, and output slice assertions. Also add a fast regression test for the `control_video_latents` path from Issue 2.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Bắt đầu với các tệp pipeline CogVideoX bị ảnh hưởng được liệt kê trong từng phần của issue, đặc biệt là pipeline_cogvideox.py và pipeline_cogvideox_fun_control.py, sau đó chạy các bản tái hiện độc lập được cung cấp bằng .venv/Scripts/python.exe. Xem xét các tiền lệ liên quan trong pipeline WAN và Mochi. Được xem là hoàn tất khi các trường hợp batch, control-latent, mixed-dtype và validation được báo cáo hoạt động chính xác, đồng thời các đường dẫn bị ảnh hưởng có coverage.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ: python, pytorch
Lĩnh vực: machine-learning
Loại issue: Lỗi
Độ khó: 4/5
Thời gian dự kiến: 3-5 ngày
Mức độ hoạt động: Ít trao đổi
Độ rõ ràng: Khá rõ ràng
Mức phù hợp với người mới: 45/100

cogvideo model/pipeline review

Mô tả

Hướng dẫn đóng góp

Hướng nghiên cứu

Đánh giá

Nhận issue mới trong hộp thư của bạn