huggingface / huggingface/diffusers
lucy model/pipeline review
- Langage dominant
- Python
- Étoiles
- 34.5k
- Forks
- 7.3k
- Merge moyen
- 3 j 3 h
- PR mergées (30 j)
- 91
Description
# `lucy` model/pipeline review
Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`
Review performed against the repository review rules.
Duplicate search status: searched GitHub issues and PRs in `huggingface/diffusers` for `lucy`, `LucyEditPipeline`, `pipeline_lucy_edit`, `LucyPipelineOutput`, `num_videos_per_prompt condition_latents`, `ftfy basic_clean prompt_clean`, and `Lucy tests`. No duplicate issue/PR found for the findings below. Existing related PRs found: original implementation PR https://github.com/huggingface/diffusers/pull/12340 and typo PR https://github.com/huggingface/diffusers/pull/12705.
## Issue 1: `num_videos_per_prompt > 1` fails because condition latents are not expanded
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L619-L629
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L403-L424
Problem:
`__call__` passes `batch_size * num_videos_per_prompt` to `prepare_latents`, so random latents and prompt embeddings are expanded. The conditioning video latents are encoded only once per input video and are never repeated, then an assertion requires them to match the expanded latent batch.
Impact:
The public `num_videos_per_prompt` argument is broken for values greater than 1.
Reproduction:
```python
from types import SimpleNamespace
import torch
from diffusers import LucyEditPipeline
class FakeVAE:
config = SimpleNamespace(
scale_factor_temporal=4,
scale_factor_spatial=8,
z_dim=16,
latents_mean=[0.0] * 16,
latents_std=[1.0] * 16,
)
def encode(self, x):
b, c, f, h, w = x.shape
latent_frames = (f - 1) // self.config.scale_factor_temporal + 1
return SimpleNamespace(latents=torch.zeros(b, 16, latent_frames, h // 8, w // 8))
pipe = object.__new__(LucyEditPipeline)
pipe.vae = FakeVAE()
pipe.vae_scale_factor_temporal = 4
pipe.vae_scale_factor_spatial = 8
video = torch.zeros(1, 3, 17, 16, 16)
LucyEditPipeline.prepare_latents(
pipe,
video=video,
batch_size=2, # one prompt, num_videos_per_prompt=2
num_channels_latents=16,
height=16,
width=16,
dtype=torch.float32,
device=torch.device("cpu"),
generator=torch.Generator().manual_seed(0),
)
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan_i2v.py#L449-L459
Suggested fix:
```python
condition_latents = torch.cat(condition_latents, dim=0).to(device=device, dtype=dtype)
if batch_size > condition_latents.shape[0]:
if batch_size % condition_latents.shape[0] != 0:
raise ValueError(
f"Cannot duplicate `video` batch size {condition_latents.shape[0]} to latent batch size {batch_size}."
)
condition_latents = condition_latents.repeat_interleave(batch_size // condition_latents.shape[0], dim=0)
```
## Issue 2: Prompt cleaning crashes when optional `ftfy` is unavailable
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L103-L117
Problem:
`ftfy` is optional, but `basic_clean()` calls `ftfy.fix_text()` unconditionally. If `ftfy` is not installed, Lucy prompt encoding raises `NameError`.
Impact:
A standard install without the optional text-cleaning dependency can import the pipeline but fails at runtime on normal prompt inputs.
Reproduction:
```python
from diffusers.pipelines.lucy import pipeline_lucy_edit as lucy
# Simulate an environment where optional dependency ftfy is not installed.
if hasattr(lucy, "ftfy"):
delattr(lucy, "ftfy")
lucy.prompt_clean("hello")
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan.py#L78-L82
Suggested fix:
```python
def basic_clean(text):
if is_ftfy_available():
text = ftfy.fix_text(text)
text = html.unescape(html.unescape(text))
return text.strip()
```
## Issue 3: `num_frames` is accepted but ignored for latent preparation
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L563-L568
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L616-L629
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L387-L389
Problem:
`__call__` validates and rounds `num_frames`, but `prepare_latents()` derives the latent frame count from `video.size(2)`. The requested `num_frames` does not control generation length and is not validated against the conditioning video length.
Impact:
Users can pass `num_frames` expecting it to control the output, but the output length follows the input video instead. This is especially confusing because the docstring says `num_frames` is “The number of frames in the generated video.”
Reproduction:
```python
from types import SimpleNamespace
import torch
from diffusers import LucyEditPipeline
class FakeVAE:
config = SimpleNamespace(
scale_factor_temporal=4,
scale_factor_spatial=8,
z_dim=16,
latents_mean=[0.0] * 16,
latents_std=[1.0] * 16,
)
def encode(self, x):
b, c, f, h, w = x.shape
latent_frames = (f - 1) // self.config.scale_factor_temporal + 1
return SimpleNamespace(latents=torch.zeros(b, 16, latent_frames, h // 8, w // 8))
pipe = object.__new__(LucyEditPipeline)
pipe.vae = FakeVAE()
pipe.vae_scale_factor_temporal = 4
pipe.vae_scale_factor_spatial = 8
for conditioning_frames in (9, 17):
video = torch.zeros(1, 3, conditioning_frames, 16, 16)
latents, _ = LucyEditPipeline.prepare_latents(
pipe,
video=video,
batch_size=1,
num_channels_latents=16,
height=16,
width=16,
dtype=torch.float32,
device=torch.device("cpu"),
generator=torch.Generator().manual_seed(0),
)
print(conditioning_frames, latents.shape[2])
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan_i2v.py#L393-L411
Suggested fix:
Either remove/reword `num_frames` for Lucy and validate that the conditioning video length is the generation length, or pass `num_frames` into `prepare_latents()` and crop/validate the conditioning video before encoding.
## Issue 4: No Lucy fast or slow tests exist
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L134-L168
Problem:
There is no `tests/pipelines/lucy/` coverage and no test file matching `*lucy*`. This leaves imports, save/load behavior, callbacks, dtype handling, batching, `num_videos_per_prompt`, and a slow smoke test untested.
Impact:
The two runtime bugs above are not covered by CI, and regressions in the newly added pipeline family can ship unnoticed. Slow tests are also missing.
Reproduction:
```python
from pathlib import Path
lucy_tests = sorted(Path("tests").rglob("*lucy*"))
print(lucy_tests)
assert lucy_tests, "No Lucy fast or slow tests found"
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/wan/test_wan_video_to_video.py#L35-L50
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/wan/test_wan.py#L185-L201
Suggested fix:
Add `tests/pipelines/lucy/test_lucy_edit.py` with tiny Wan components using `WanTransformer3DModel(in_channels=32, out_channels=16)`, a fast inference test, save/load coverage, callback coverage, `num_videos_per_prompt=2`, and a slow test for `decart-ai/Lucy-Edit-Dev`.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Start with src/diffusers/pipelines/lucy/pipeline_lucy_edit.py, especially prepare_latents(), prompt_clean(), and __call__(), then compare the cited Wan pipeline implementations. Add tests/pipelines/lucy/test_lucy_edit.py covering batching, optional ftfy, num_frames, save/load, callbacks, and fast/slow inference. Done means the reported behaviors are defined and covered by passing tests.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- python, pytorch
- Domaine
- machine-learning, testing-qa
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- Calme
- Clarté
- Clairement spécifiée
- Accessibilité débutants
- 45/100