huggingface / huggingface/diffusers

lucy model/pipeline review

Đang mở
#13,632 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
34.5k
Fork
7.3k
Merge trung bình
3 ngày 3 giờ
Pull request đã merge (30 ngày)
91

Mô tả

# `lucy` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

Duplicate search status: searched GitHub issues and PRs in `huggingface/diffusers` for `lucy`, `LucyEditPipeline`, `pipeline_lucy_edit`, `LucyPipelineOutput`, `num_videos_per_prompt condition_latents`, `ftfy basic_clean prompt_clean`, and `Lucy tests`. No duplicate issue/PR found for the findings below. Existing related PRs found: original implementation PR https://github.com/huggingface/diffusers/pull/12340 and typo PR https://github.com/huggingface/diffusers/pull/12705.

## Issue 1: `num_videos_per_prompt > 1` fails because condition latents are not expanded

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L619-L629
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L403-L424

Problem:
`__call__` passes `batch_size * num_videos_per_prompt` to `prepare_latents`, so random latents and prompt embeddings are expanded. The conditioning video latents are encoded only once per input video and are never repeated, then an assertion requires them to match the expanded latent batch.

Impact:
The public `num_videos_per_prompt` argument is broken for values greater than 1.

Reproduction:
```python
from types import SimpleNamespace
import torch
from diffusers import LucyEditPipeline

class FakeVAE:
config = SimpleNamespace(
scale_factor_temporal=4,
scale_factor_spatial=8,
z_dim=16,
latents_mean=[0.0] * 16,
latents_std=[1.0] * 16,
)
def encode(self, x):
b, c, f, h, w = x.shape
latent_frames = (f - 1) // self.config.scale_factor_temporal + 1
return SimpleNamespace(latents=torch.zeros(b, 16, latent_frames, h // 8, w // 8))

pipe = object.__new__(LucyEditPipeline)
pipe.vae = FakeVAE()
pipe.vae_scale_factor_temporal = 4
pipe.vae_scale_factor_spatial = 8

video = torch.zeros(1, 3, 17, 16, 16)
LucyEditPipeline.prepare_latents(
pipe,
video=video,
batch_size=2, # one prompt, num_videos_per_prompt=2
num_channels_latents=16,
height=16,
width=16,
dtype=torch.float32,
device=torch.device("cpu"),
generator=torch.Generator().manual_seed(0),
)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan_i2v.py#L449-L459

Suggested fix:
```python
condition_latents = torch.cat(condition_latents, dim=0).to(device=device, dtype=dtype)

if batch_size > condition_latents.shape[0]:
if batch_size % condition_latents.shape[0] != 0:
raise ValueError(
f"Cannot duplicate `video` batch size {condition_latents.shape[0]} to latent batch size {batch_size}."
)
condition_latents = condition_latents.repeat_interleave(batch_size // condition_latents.shape[0], dim=0)
```

## Issue 2: Prompt cleaning crashes when optional `ftfy` is unavailable

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L103-L117

Problem:
`ftfy` is optional, but `basic_clean()` calls `ftfy.fix_text()` unconditionally. If `ftfy` is not installed, Lucy prompt encoding raises `NameError`.

Impact:
A standard install without the optional text-cleaning dependency can import the pipeline but fails at runtime on normal prompt inputs.

Reproduction:
```python
from diffusers.pipelines.lucy import pipeline_lucy_edit as lucy

# Simulate an environment where optional dependency ftfy is not installed.
if hasattr(lucy, "ftfy"):
delattr(lucy, "ftfy")

lucy.prompt_clean("hello")
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan.py#L78-L82

Suggested fix:
```python
def basic_clean(text):
if is_ftfy_available():
text = ftfy.fix_text(text)
text = html.unescape(html.unescape(text))
return text.strip()
```

## Issue 3: `num_frames` is accepted but ignored for latent preparation

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L563-L568
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L616-L629
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L387-L389

Problem:
`__call__` validates and rounds `num_frames`, but `prepare_latents()` derives the latent frame count from `video.size(2)`. The requested `num_frames` does not control generation length and is not validated against the conditioning video length.

Impact:
Users can pass `num_frames` expecting it to control the output, but the output length follows the input video instead. This is especially confusing because the docstring says `num_frames` is “The number of frames in the generated video.”

Reproduction:
```python
from types import SimpleNamespace
import torch
from diffusers import LucyEditPipeline

class FakeVAE:
config = SimpleNamespace(
scale_factor_temporal=4,
scale_factor_spatial=8,
z_dim=16,
latents_mean=[0.0] * 16,
latents_std=[1.0] * 16,
)
def encode(self, x):
b, c, f, h, w = x.shape
latent_frames = (f - 1) // self.config.scale_factor_temporal + 1
return SimpleNamespace(latents=torch.zeros(b, 16, latent_frames, h // 8, w // 8))

pipe = object.__new__(LucyEditPipeline)
pipe.vae = FakeVAE()
pipe.vae_scale_factor_temporal = 4
pipe.vae_scale_factor_spatial = 8

for conditioning_frames in (9, 17):
video = torch.zeros(1, 3, conditioning_frames, 16, 16)
latents, _ = LucyEditPipeline.prepare_latents(
pipe,
video=video,
batch_size=1,
num_channels_latents=16,
height=16,
width=16,
dtype=torch.float32,
device=torch.device("cpu"),
generator=torch.Generator().manual_seed(0),
)
print(conditioning_frames, latents.shape[2])
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan_i2v.py#L393-L411

Suggested fix:
Either remove/reword `num_frames` for Lucy and validate that the conditioning video length is the generation length, or pass `num_frames` into `prepare_latents()` and crop/validate the conditioning video before encoding.

## Issue 4: No Lucy fast or slow tests exist

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L134-L168

Problem:
There is no `tests/pipelines/lucy/` coverage and no test file matching `*lucy*`. This leaves imports, save/load behavior, callbacks, dtype handling, batching, `num_videos_per_prompt`, and a slow smoke test untested.

Impact:
The two runtime bugs above are not covered by CI, and regressions in the newly added pipeline family can ship unnoticed. Slow tests are also missing.

Reproduction:
```python
from pathlib import Path

lucy_tests = sorted(Path("tests").rglob("*lucy*"))
print(lucy_tests)
assert lucy_tests, "No Lucy fast or slow tests found"
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/wan/test_wan_video_to_video.py#L35-L50
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/wan/test_wan.py#L185-L201

Suggested fix:
Add `tests/pipelines/lucy/test_lucy_edit.py` with tiny Wan components using `WanTransformer3DModel(in_channels=32, out_channels=16)`, a fast inference test, save/load coverage, callback coverage, `num_videos_per_prompt=2`, and a slow test for `decart-ai/Lucy-Edit-Dev`.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Bắt đầu với src/diffusers/pipelines/lucy/pipeline_lucy_edit.py, đặc biệt là prepare_latents(), prompt_clean() và __call__(), sau đó so sánh với các triển khai pipeline Wan được trích dẫn. Thêm tests/pipelines/lucy/test_lucy_edit.py để bao phủ xử lý theo lô, ftfy tùy chọn, num_frames, lưu/tải, callbacks và suy luận nhanh/chậm. Công việc được xem là hoàn tất khi các hành vi được báo cáo đã được định nghĩa và được bao phủ bởi các bài kiểm thử đạt.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python, pytorch
Lĩnh vực
machine-learning, testing-qa
Loại issue
Lỗi
Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Đặc tả rõ ràng
Mức phù hợp với người mới
45/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.