huggingface / huggingface/diffusers

lucy model/pipeline review

Open
#13,632 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

# `lucy` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

Duplicate search status: searched GitHub issues and PRs in `huggingface/diffusers` for `lucy`, `LucyEditPipeline`, `pipeline_lucy_edit`, `LucyPipelineOutput`, `num_videos_per_prompt condition_latents`, `ftfy basic_clean prompt_clean`, and `Lucy tests`. No duplicate issue/PR found for the findings below. Existing related PRs found: original implementation PR https://github.com/huggingface/diffusers/pull/12340 and typo PR https://github.com/huggingface/diffusers/pull/12705.

## Issue 1: `num_videos_per_prompt > 1` fails because condition latents are not expanded

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L619-L629
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L403-L424

Problem:
`__call__` passes `batch_size * num_videos_per_prompt` to `prepare_latents`, so random latents and prompt embeddings are expanded. The conditioning video latents are encoded only once per input video and are never repeated, then an assertion requires them to match the expanded latent batch.

Impact:
The public `num_videos_per_prompt` argument is broken for values greater than 1.

Reproduction:
```python
from types import SimpleNamespace
import torch
from diffusers import LucyEditPipeline

class FakeVAE:
config = SimpleNamespace(
scale_factor_temporal=4,
scale_factor_spatial=8,
z_dim=16,
latents_mean=[0.0] * 16,
latents_std=[1.0] * 16,
)
def encode(self, x):
b, c, f, h, w = x.shape
latent_frames = (f - 1) // self.config.scale_factor_temporal + 1
return SimpleNamespace(latents=torch.zeros(b, 16, latent_frames, h // 8, w // 8))

pipe = object.__new__(LucyEditPipeline)
pipe.vae = FakeVAE()
pipe.vae_scale_factor_temporal = 4
pipe.vae_scale_factor_spatial = 8

video = torch.zeros(1, 3, 17, 16, 16)
LucyEditPipeline.prepare_latents(
pipe,
video=video,
batch_size=2, # one prompt, num_videos_per_prompt=2
num_channels_latents=16,
height=16,
width=16,
dtype=torch.float32,
device=torch.device("cpu"),
generator=torch.Generator().manual_seed(0),
)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan_i2v.py#L449-L459

Suggested fix:
```python
condition_latents = torch.cat(condition_latents, dim=0).to(device=device, dtype=dtype)

if batch_size > condition_latents.shape[0]:
if batch_size % condition_latents.shape[0] != 0:
raise ValueError(
f"Cannot duplicate `video` batch size {condition_latents.shape[0]} to latent batch size {batch_size}."
)
condition_latents = condition_latents.repeat_interleave(batch_size // condition_latents.shape[0], dim=0)
```

## Issue 2: Prompt cleaning crashes when optional `ftfy` is unavailable

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L103-L117

Problem:
`ftfy` is optional, but `basic_clean()` calls `ftfy.fix_text()` unconditionally. If `ftfy` is not installed, Lucy prompt encoding raises `NameError`.

Impact:
A standard install without the optional text-cleaning dependency can import the pipeline but fails at runtime on normal prompt inputs.

Reproduction:
```python
from diffusers.pipelines.lucy import pipeline_lucy_edit as lucy

# Simulate an environment where optional dependency ftfy is not installed.
if hasattr(lucy, "ftfy"):
delattr(lucy, "ftfy")

lucy.prompt_clean("hello")
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan.py#L78-L82

Suggested fix:
```python
def basic_clean(text):
if is_ftfy_available():
text = ftfy.fix_text(text)
text = html.unescape(html.unescape(text))
return text.strip()
```

## Issue 3: `num_frames` is accepted but ignored for latent preparation

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L563-L568
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L616-L629
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L387-L389

Problem:
`__call__` validates and rounds `num_frames`, but `prepare_latents()` derives the latent frame count from `video.size(2)`. The requested `num_frames` does not control generation length and is not validated against the conditioning video length.

Impact:
Users can pass `num_frames` expecting it to control the output, but the output length follows the input video instead. This is especially confusing because the docstring says `num_frames` is “The number of frames in the generated video.”

Reproduction:
```python
from types import SimpleNamespace
import torch
from diffusers import LucyEditPipeline

class FakeVAE:
config = SimpleNamespace(
scale_factor_temporal=4,
scale_factor_spatial=8,
z_dim=16,
latents_mean=[0.0] * 16,
latents_std=[1.0] * 16,
)
def encode(self, x):
b, c, f, h, w = x.shape
latent_frames = (f - 1) // self.config.scale_factor_temporal + 1
return SimpleNamespace(latents=torch.zeros(b, 16, latent_frames, h // 8, w // 8))

pipe = object.__new__(LucyEditPipeline)
pipe.vae = FakeVAE()
pipe.vae_scale_factor_temporal = 4
pipe.vae_scale_factor_spatial = 8

for conditioning_frames in (9, 17):
video = torch.zeros(1, 3, conditioning_frames, 16, 16)
latents, _ = LucyEditPipeline.prepare_latents(
pipe,
video=video,
batch_size=1,
num_channels_latents=16,
height=16,
width=16,
dtype=torch.float32,
device=torch.device("cpu"),
generator=torch.Generator().manual_seed(0),
)
print(conditioning_frames, latents.shape[2])
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan_i2v.py#L393-L411

Suggested fix:
Either remove/reword `num_frames` for Lucy and validate that the conditioning video length is the generation length, or pass `num_frames` into `prepare_latents()` and crop/validate the conditioning video before encoding.

## Issue 4: No Lucy fast or slow tests exist

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/lucy/pipeline_lucy_edit.py#L134-L168

Problem:
There is no `tests/pipelines/lucy/` coverage and no test file matching `*lucy*`. This leaves imports, save/load behavior, callbacks, dtype handling, batching, `num_videos_per_prompt`, and a slow smoke test untested.

Impact:
The two runtime bugs above are not covered by CI, and regressions in the newly added pipeline family can ship unnoticed. Slow tests are also missing.

Reproduction:
```python
from pathlib import Path

lucy_tests = sorted(Path("tests").rglob("*lucy*"))
print(lucy_tests)
assert lucy_tests, "No Lucy fast or slow tests found"
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/wan/test_wan_video_to_video.py#L35-L50
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/wan/test_wan.py#L185-L201

Suggested fix:
Add `tests/pipelines/lucy/test_lucy_edit.py` with tiny Wan components using `WanTransformer3DModel(in_channels=32, out_channels=16)`, a fast inference test, save/load coverage, callback coverage, `num_videos_per_prompt=2`, and a slow test for `decart-ai/Lucy-Edit-Dev`.

Contributor guide

Open the contributing guide

Research direction

Start with src/diffusers/pipelines/lucy/pipeline_lucy_edit.py, especially prepare_latents(), prompt_clean(), and __call__(), then compare the cited Wan pipeline implementations. Add tests/pipelines/lucy/test_lucy_edit.py covering batching, optional ftfy, num_frames, save/load, callbacks, and fast/slow inference. Done means the reported behaviors are defined and covered by passing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.