huggingface / huggingface/diffusers

easyanimate model/pipeline review

Aperta
#13,638 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

# `easyanimate` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

Duplicate search: checked GitHub Issues/PRs for `EasyAnimate`, affected class/function names, and the specific failure modes. No exact duplicates found. Related: https://github.com/huggingface/diffusers/issues/12646 reports another crash in the same inpaint repaint branch; https://github.com/huggingface/diffusers/pull/13347 only refactors transformer tests.

Local test note: `.venv` was used. Focused Python reproductions ran; full non-slow pytest collection is blocked in this `.venv` by `ModuleNotFoundError: torch._C._distributed_c10d`.

## Issue 1: `EasyAnimateControlPipeline.__call__` always passes an invalid `encode_prompt` kwarg

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_control.py#L796-L808

Problem:
`EasyAnimateControlPipeline.encode_prompt()` does not accept `text_encoder_index`, but `__call__` passes `text_encoder_index=0`. Any control pipeline call reaches this TypeError before denoising.

Impact:
`EasyAnimateControlPipeline` is effectively unusable through its public `__call__`.

Reproduction:
```python
from diffusers import EasyAnimateControlPipeline

pipe = object.__new__(EasyAnimateControlPipeline)
EasyAnimateControlPipeline.encode_prompt(pipe, prompt="x", text_encoder_index=0)
```

Relevant precedent:
Base and inpaint EasyAnimate call `encode_prompt` without this kwarg:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate.py#L653-L664

Suggested fix:
```python
# Remove the stale kwarg from EasyAnimateControlPipeline.__call__
negative_prompt_attention_mask=negative_prompt_attention_mask,
```

## Issue 2: `EasyAnimateControlPipeline` decodes through an undefined method

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_control.py#L985-L988

Problem:
The control pipeline calls `self.decode_latents(latents)`, but the class does not define `decode_latents`.

Impact:
After fixing Issue 1, any control run with `output_type != "latent"` will fail at decode time.

Reproduction:
```python
from diffusers import EasyAnimateControlPipeline

pipe = object.__new__(EasyAnimateControlPipeline)
print(hasattr(pipe, "decode_latents"))
pipe.decode_latents(None)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate.py#L761-L764

Suggested fix:
```python
latents = 1 / self.vae.config.scaling_factor * latents
video = self.vae.decode(latents, return_dict=False)[0]
video = self.video_processor.postprocess_video(video=video, output_type=output_type)
```

## Issue 3: Control helper optional mask/reference paths are broken

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_control.py#L120-L149

Problem:
`get_video_to_video_latent()` calls `preprocess_image(..., size=sample_size)`, but the helper parameter is named `sample_size`. The `ref_image` branch also builds the wrong rank for the pipeline, which later expects `(B, C, F, H, W)`.

Impact:
Documented/public helper paths for control masks and reference images fail before pipeline execution.

Reproduction:
```python
from PIL import Image
from diffusers.pipelines.easyanimate.pipeline_easyanimate_control import get_video_to_video_latent

frame = Image.new("RGB", (8, 8), "white")
mask = Image.new("RGB", (8, 8), "black")
ref = Image.new("RGB", (8, 8), "blue")

for kwargs in ({"validation_video_mask": mask}, {"ref_image": ref}):
try:
get_video_to_video_latent([frame], 1, (8, 8), **kwargs)
except Exception as e:
print(type(e).__name__, e)
```

Relevant precedent:
The main video path already uses `sample_size=sample_size` at line 123.

Suggested fix:
```python
validation_video_mask = preprocess_image(validation_video_mask, sample_size=sample_size)[:1]
input_video_mask = torch.where(validation_video_mask < 240 / 255.0, 0.0, 255.0)
input_video_mask = input_video_mask.unsqueeze(0).unsqueeze(2).repeat(1, 1, input_video.shape[2], 1, 1)

ref_image = preprocess_image(ref_image, sample_size=sample_size)
ref_image = ref_image.unsqueeze(1).unsqueeze(0)
```

## Issue 4: Inpaint helper fails for multiple end frames

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_inpaint.py#L156-L164

Problem:
For `validation_image_end` as a list, the code slices with `len(end_video)`, which is the batch dimension (`1`), not the number of end frames.

Impact:
Multi-frame end conditioning raises a shape error and cannot prepare inputs.

Reproduction:
```python
from PIL import Image
from diffusers.pipelines.easyanimate.pipeline_easyanimate_inpaint import get_image_to_video_latent

start = Image.new("RGB", (8, 8), "white")
ends = [Image.new("RGB", (8, 8), "black"), Image.new("RGB", (8, 8), "blue")]
get_image_to_video_latent(start, ends, 4, (8, 8))
```

Relevant precedent:
The mask branch already uses `len(image_end)` on the next line.

Suggested fix:
```python
input_video[:, :, -len(image_end) :] = end_video
input_video_mask[:, :, -len(image_end) :] = 0
```

## Issue 5: Inpaint FlowMatch repaint branch has a malformed `torch.tensor` call

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_inpaint.py#L1194-L1202

Problem:
`torch.tensor([noise_timestep], noise)` passes `noise` as a second positional argument to `torch.tensor`, which is invalid. It also fails to pass `noise` to `scale_noise`.

Impact:
The repaint branch for `FlowMatchEulerDiscreteScheduler` crashes when `num_channels_transformer == num_channels_latents`.

Reproduction:
```python
import torch
from diffusers import FlowMatchEulerDiscreteScheduler

scheduler = FlowMatchEulerDiscreteScheduler()
scheduler.set_timesteps(2)

sample = torch.zeros(1, 4, 1, 2, 2)
noise = torch.ones_like(sample)
noise_timestep = scheduler.timesteps[1]

scheduler.scale_noise(sample, torch.tensor([noise_timestep], noise))
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/qwenimage/pipeline_qwenimage_inpaint.py#L1003-L1007

Suggested fix:
```python
init_latents_proper = self.scheduler.scale_noise(
init_latents_proper, torch.tensor([noise_timestep], device=device), noise
)
```

## Issue 6: EasyAnimate attention ignores Diffusers attention backends

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_easyanimate.py#L112-L182

Problem:
`EasyAnimateAttnProcessor2_0` calls `F.scaled_dot_product_attention` directly and does not define `_attention_backend` / `_parallel_config`. `model.set_attention_backend(...)` silently skips the processor.

Impact:
Users cannot select supported attention backends for EasyAnimate, and context-parallel/backend plumbing cannot affect this model.

Reproduction:
```python
from diffusers import EasyAnimateTransformer3DModel

model = EasyAnimateTransformer3DModel(
num_attention_heads=2,
attention_head_dim=16,
in_channels=4,
out_channels=4,
time_embed_dim=8,
text_embed_dim=16,
num_layers=1,
mmdit_layers=1,
patch_size=2,
)

processor = model.transformer_blocks[0].attn1.processor
print(hasattr(processor, "_attention_backend"))
model.set_attention_backend("native")
print(getattr(processor, "_attention_backend", None))
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_flux.py#L75-L125

Suggested fix:
```python
from ..attention_dispatch import dispatch_attention_fn

class EasyAnimateAttnProcessor2_0:
_attention_backend = None
_parallel_config = None

...
hidden_states = dispatch_attention_fn(
query,
key,
value,
attn_mask=attention_mask,
dropout_p=0.0,
is_causal=False,
backend=self._attention_backend,
parallel_config=self._parallel_config,
)
```

## Issue 7: Control and inpaint variants have no fast or slow pipeline coverage

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/easyanimate/test_easyanimate.py#L45-L46
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/easyanimate/test_easyanimate.py#L260-L296
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/docs/source/en/api/pipelines/easyanimate.md#L80-L88

Problem:
The test file only covers `EasyAnimatePipeline`; there are no fast tests or slow/integration tests for `EasyAnimateControlPipeline`, `EasyAnimateInpaintPipeline`, or their public helper functions. Docs also only autodoc the base pipeline.

Impact:
The concrete control/inpaint regressions above are not caught by CI, and public variants are less discoverable.

Reproduction:
```python
from pathlib import Path

tests = Path("tests/pipelines/easyanimate/test_easyanimate.py").read_text()
docs = Path("docs/source/en/api/pipelines/easyanimate.md").read_text()

print("EasyAnimateControlPipeline" in tests)
print("EasyAnimateInpaintPipeline" in tests)
print("[[autodoc]] EasyAnimateControlPipeline" in docs)
print("[[autodoc]] EasyAnimateInpaintPipeline" in docs)
```

Relevant precedent:
Existing base pipeline fast and slow tests are in the same file and can be extended with tiny control/inpaint fixtures.

Suggested fix:
Add focused fast tests for control and inpaint using tiny components, including `output_type="pt"` and `"latent"` paths plus helper utility tests. Add slow tests for official control and inpaint checkpoints, or explicitly mark/document why they cannot be run. Update the EasyAnimate docs to autodoc `EasyAnimateControlPipeline` and `EasyAnimateInpaintPipeline`.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start with the affected EasyAnimate pipeline and transformer files named in the issue, then run the focused Python reproductions for the control, inpaint, and attention paths. Extend tests/pipelines/easyanimate/test_easyanimate.py and the EasyAnimate API docs as described, verifying the regressions and public variants; full pytest collection is currently blocked by the reported torch._C._distributed_c10d error.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
documentation, machine-learning, testing-qa
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Specificata chiaramente
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.