huggingface / huggingface/diffusers

lucy model/pipeline review

オープン
#13,632 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
34.5k
フォーク
7.3k
平均マージ
3日 3時間
マージ済み PR(30日)
91

説明

# `lucy` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

Duplicate search status: searched GitHub issues and PRs in `huggingface/diffusers` for `lucy`, `LucyEditPipeline`, `pipeline_lucy_edit`, `LucyPipelineOutput`, `num_videos_per_prompt condition_latents`, `ftfy basic_clean prompt_clean`, and `Lucy tests`. No duplicate issue/PR found for the findings below. Existing related PRs found: original implementation PR https://github.com/huggingface/diffusers/pull/12340 and typo PR https://github.com/huggingface/diffusers/pull/12705.

## Issue 1: `num_videos_per_prompt > 1` fails because condition latents are not expanded

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L619-L629
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L403-L424

Problem:
`__call__` passes `batch_size * num_videos_per_prompt` to `prepare_latents`, so random latents and prompt embeddings are expanded. The conditioning video latents are encoded only once per input video and are never repeated, then an assertion requires them to match the expanded latent batch.

Impact:
The public `num_videos_per_prompt` argument is broken for values greater than 1.

Reproduction:
```python
from types import SimpleNamespace
import torch
from diffusers import LucyEditPipeline

class FakeVAE:
config = SimpleNamespace(
scale_factor_temporal=4,
scale_factor_spatial=8,
z_dim=16,
latents_mean=[0.0] * 16,
latents_std=[1.0] * 16,
)
def encode(self, x):
b, c, f, h, w = x.shape
latent_frames = (f - 1) // self.config.scale_factor_temporal + 1
return SimpleNamespace(latents=torch.zeros(b, 16, latent_frames, h // 8, w // 8))

pipe = object.__new__(LucyEditPipeline)
pipe.vae = FakeVAE()
pipe.vae_scale_factor_temporal = 4
pipe.vae_scale_factor_spatial = 8

video = torch.zeros(1, 3, 17, 16, 16)
LucyEditPipeline.prepare_latents(
pipe,
video=video,
batch_size=2, # one prompt, num_videos_per_prompt=2
num_channels_latents=16,
height=16,
width=16,
dtype=torch.float32,
device=torch.device("cpu"),
generator=torch.Generator().manual_seed(0),
)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan_i2v.py#L449-L459

Suggested fix:
```python
condition_latents = torch.cat(condition_latents, dim=0).to(device=device, dtype=dtype)

if batch_size > condition_latents.shape[0]:
if batch_size % condition_latents.shape[0] != 0:
raise ValueError(
f"Cannot duplicate `video` batch size {condition_latents.shape[0]} to latent batch size {batch_size}."
)
condition_latents = condition_latents.repeat_interleave(batch_size // condition_latents.shape[0], dim=0)
```

## Issue 2: Prompt cleaning crashes when optional `ftfy` is unavailable

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L103-L117

Problem:
`ftfy` is optional, but `basic_clean()` calls `ftfy.fix_text()` unconditionally. If `ftfy` is not installed, Lucy prompt encoding raises `NameError`.

Impact:
A standard install without the optional text-cleaning dependency can import the pipeline but fails at runtime on normal prompt inputs.

Reproduction:
```python
from diffusers.pipelines.lucy import pipeline_lucy_edit as lucy

# Simulate an environment where optional dependency ftfy is not installed.
if hasattr(lucy, "ftfy"):
delattr(lucy, "ftfy")

lucy.prompt_clean("hello")
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan.py#L78-L82

Suggested fix:
```python
def basic_clean(text):
if is_ftfy_available():
text = ftfy.fix_text(text)
text = html.unescape(html.unescape(text))
return text.strip()
```

## Issue 3: `num_frames` is accepted but ignored for latent preparation

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L563-L568
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L616-L629
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L387-L389

Problem:
`__call__` validates and rounds `num_frames`, but `prepare_latents()` derives the latent frame count from `video.size(2)`. The requested `num_frames` does not control generation length and is not validated against the conditioning video length.

Impact:
Users can pass `num_frames` expecting it to control the output, but the output length follows the input video instead. This is especially confusing because the docstring says `num_frames` is “The number of frames in the generated video.”

Reproduction:
```python
from types import SimpleNamespace
import torch
from diffusers import LucyEditPipeline

class FakeVAE:
config = SimpleNamespace(
scale_factor_temporal=4,
scale_factor_spatial=8,
z_dim=16,
latents_mean=[0.0] * 16,
latents_std=[1.0] * 16,
)
def encode(self, x):
b, c, f, h, w = x.shape
latent_frames = (f - 1) // self.config.scale_factor_temporal + 1
return SimpleNamespace(latents=torch.zeros(b, 16, latent_frames, h // 8, w // 8))

pipe = object.__new__(LucyEditPipeline)
pipe.vae = FakeVAE()
pipe.vae_scale_factor_temporal = 4
pipe.vae_scale_factor_spatial = 8

for conditioning_frames in (9, 17):
video = torch.zeros(1, 3, conditioning_frames, 16, 16)
latents, _ = LucyEditPipeline.prepare_latents(
pipe,
video=video,
batch_size=1,
num_channels_latents=16,
height=16,
width=16,
dtype=torch.float32,
device=torch.device("cpu"),
generator=torch.Generator().manual_seed(0),
)
print(conditioning_frames, latents.shape[2])
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan_i2v.py#L393-L411

Suggested fix:
Either remove/reword `num_frames` for Lucy and validate that the conditioning video length is the generation length, or pass `num_frames` into `prepare_latents()` and crop/validate the conditioning video before encoding.

## Issue 4: No Lucy fast or slow tests exist

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L134-L168

Problem:
There is no `tests/pipelines/lucy/` coverage and no test file matching `*lucy*`. This leaves imports, save/load behavior, callbacks, dtype handling, batching, `num_videos_per_prompt`, and a slow smoke test untested.

Impact:
The two runtime bugs above are not covered by CI, and regressions in the newly added pipeline family can ship unnoticed. Slow tests are also missing.

Reproduction:
```python
from pathlib import Path

lucy_tests = sorted(Path("tests").rglob("*lucy*"))
print(lucy_tests)
assert lucy_tests, "No Lucy fast or slow tests found"
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/wan/test_wan_video_to_video.py#L35-L50
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/wan/test_wan.py#L185-L201

Suggested fix:
Add `tests/pipelines/lucy/test_lucy_edit.py` with tiny Wan components using `WanTransformer3DModel(in_channels=32, out_channels=16)`, a fast inference test, save/load coverage, callback coverage, `num_videos_per_prompt=2`, and a slow test for `decart-ai/Lucy-Edit-Dev`.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start with src/diffusers/pipelines/lucy/pipeline_lucy_edit.py, especially prepare_latents(), prompt_clean(), and __call__(), then compare the cited Wan pipeline implementations. Add tests/pipelines/lucy/test_lucy_edit.py covering batching, optional ftfy, num_frames, save/load, callbacks, and fast/slow inference. Done means the reported behaviors are defined and covered by passing tests.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python, pytorch
領域
machine-learning, testing-qa
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
静か
明瞭さ
明確に書かれている
初心者へのやさしさ
45/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。