huggingface / huggingface/diffusers
lucy model/pipeline review
- Ngôn ngữ chính
- Python
- Star
- 34.5k
- Fork
- 7.3k
- Merge trung bình
- 3 ngày 3 giờ
- Pull request đã merge (30 ngày)
- 91
Mô tả
# `lucy` model/pipeline review
Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`
Review performed against the repository review rules.
Duplicate search status: searched GitHub issues and PRs in `huggingface/diffusers` for `lucy`, `LucyEditPipeline`, `pipeline_lucy_edit`, `LucyPipelineOutput`, `num_videos_per_prompt condition_latents`, `ftfy basic_clean prompt_clean`, and `Lucy tests`. No duplicate issue/PR found for the findings below. Existing related PRs found: original implementation PR https://github.com/huggingface/diffusers/pull/12340 and typo PR https://github.com/huggingface/diffusers/pull/12705.
## Issue 1: `num_videos_per_prompt > 1` fails because condition latents are not expanded
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L619-L629
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L403-L424
Problem:
`__call__` passes `batch_size * num_videos_per_prompt` to `prepare_latents`, so random latents and prompt embeddings are expanded. The conditioning video latents are encoded only once per input video and are never repeated, then an assertion requires them to match the expanded latent batch.
Impact:
The public `num_videos_per_prompt` argument is broken for values greater than 1.
Reproduction:
```python
from types import SimpleNamespace
import torch
from diffusers import LucyEditPipeline
class FakeVAE:
config = SimpleNamespace(
scale_factor_temporal=4,
scale_factor_spatial=8,
z_dim=16,
latents_mean=[0.0] * 16,
latents_std=[1.0] * 16,
)
def encode(self, x):
b, c, f, h, w = x.shape
latent_frames = (f - 1) // self.config.scale_factor_temporal + 1
return SimpleNamespace(latents=torch.zeros(b, 16, latent_frames, h // 8, w // 8))
pipe = object.__new__(LucyEditPipeline)
pipe.vae = FakeVAE()
pipe.vae_scale_factor_temporal = 4
pipe.vae_scale_factor_spatial = 8
video = torch.zeros(1, 3, 17, 16, 16)
LucyEditPipeline.prepare_latents(
pipe,
video=video,
batch_size=2, # one prompt, num_videos_per_prompt=2
num_channels_latents=16,
height=16,
width=16,
dtype=torch.float32,
device=torch.device("cpu"),
generator=torch.Generator().manual_seed(0),
)
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan_i2v.py#L449-L459
Suggested fix:
```python
condition_latents = torch.cat(condition_latents, dim=0).to(device=device, dtype=dtype)
if batch_size > condition_latents.shape[0]:
if batch_size % condition_latents.shape[0] != 0:
raise ValueError(
f"Cannot duplicate `video` batch size {condition_latents.shape[0]} to latent batch size {batch_size}."
)
condition_latents = condition_latents.repeat_interleave(batch_size // condition_latents.shape[0], dim=0)
```
## Issue 2: Prompt cleaning crashes when optional `ftfy` is unavailable
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L103-L117
Problem:
`ftfy` is optional, but `basic_clean()` calls `ftfy.fix_text()` unconditionally. If `ftfy` is not installed, Lucy prompt encoding raises `NameError`.
Impact:
A standard install without the optional text-cleaning dependency can import the pipeline but fails at runtime on normal prompt inputs.
Reproduction:
```python
from diffusers.pipelines.lucy import pipeline_lucy_edit as lucy
# Simulate an environment where optional dependency ftfy is not installed.
if hasattr(lucy, "ftfy"):
delattr(lucy, "ftfy")
lucy.prompt_clean("hello")
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan.py#L78-L82
Suggested fix:
```python
def basic_clean(text):
if is_ftfy_available():
text = ftfy.fix_text(text)
text = html.unescape(html.unescape(text))
return text.strip()
```
## Issue 3: `num_frames` is accepted but ignored for latent preparation
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L563-L568
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L616-L629
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L387-L389
Problem:
`__call__` validates and rounds `num_frames`, but `prepare_latents()` derives the latent frame count from `video.size(2)`. The requested `num_frames` does not control generation length and is not validated against the conditioning video length.
Impact:
Users can pass `num_frames` expecting it to control the output, but the output length follows the input video instead. This is especially confusing because the docstring says `num_frames` is “The number of frames in the generated video.”
Reproduction:
```python
from types import SimpleNamespace
import torch
from diffusers import LucyEditPipeline
class FakeVAE:
config = SimpleNamespace(
scale_factor_temporal=4,
scale_factor_spatial=8,
z_dim=16,
latents_mean=[0.0] * 16,
latents_std=[1.0] * 16,
)
def encode(self, x):
b, c, f, h, w = x.shape
latent_frames = (f - 1) // self.config.scale_factor_temporal + 1
return SimpleNamespace(latents=torch.zeros(b, 16, latent_frames, h // 8, w // 8))
pipe = object.__new__(LucyEditPipeline)
pipe.vae = FakeVAE()
pipe.vae_scale_factor_temporal = 4
pipe.vae_scale_factor_spatial = 8
for conditioning_frames in (9, 17):
video = torch.zeros(1, 3, conditioning_frames, 16, 16)
latents, _ = LucyEditPipeline.prepare_latents(
pipe,
video=video,
batch_size=1,
num_channels_latents=16,
height=16,
width=16,
dtype=torch.float32,
device=torch.device("cpu"),
generator=torch.Generator().manual_seed(0),
)
print(conditioning_frames, latents.shape[2])
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan_i2v.py#L393-L411
Suggested fix:
Either remove/reword `num_frames` for Lucy and validate that the conditioning video length is the generation length, or pass `num_frames` into `prepare_latents()` and crop/validate the conditioning video before encoding.
## Issue 4: No Lucy fast or slow tests exist
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L134-L168
Problem:
There is no `tests/pipelines/lucy/` coverage and no test file matching `*lucy*`. This leaves imports, save/load behavior, callbacks, dtype handling, batching, `num_videos_per_prompt`, and a slow smoke test untested.
Impact:
The two runtime bugs above are not covered by CI, and regressions in the newly added pipeline family can ship unnoticed. Slow tests are also missing.
Reproduction:
```python
from pathlib import Path
lucy_tests = sorted(Path("tests").rglob("*lucy*"))
print(lucy_tests)
assert lucy_tests, "No Lucy fast or slow tests found"
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/wan/test_wan_video_to_video.py#L35-L50
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/wan/test_wan.py#L185-L201
Suggested fix:
Add `tests/pipelines/lucy/test_lucy_edit.py` with tiny Wan components using `WanTransformer3DModel(in_channels=32, out_channels=16)`, a fast inference test, save/load coverage, callback coverage, `num_videos_per_prompt=2`, and a slow test for `decart-ai/Lucy-Edit-Dev`.
Hướng dẫn đóng góp
Hướng nghiên cứu
Bắt đầu với src/diffusers/pipelines/lucy/pipeline_lucy_edit.py, đặc biệt là prepare_latents(), prompt_clean() và __call__(), sau đó so sánh với các triển khai pipeline Wan được trích dẫn. Thêm tests/pipelines/lucy/test_lucy_edit.py để bao phủ xử lý theo lô, ftfy tùy chọn, num_frames, lưu/tải, callbacks và suy luận nhanh/chậm. Công việc được xem là hoàn tất khi các hành vi được báo cáo đã được định nghĩa và được bao phủ bởi các bài kiểm thử đạt.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- python, pytorch
- Lĩnh vực
- machine-learning, testing-qa
- Loại issue
- Lỗi
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức độ hoạt động
- Ít trao đổi
- Độ rõ ràng
- Đặc tả rõ ràng
- Mức phù hợp với người mới
- 45/100