huggingface / huggingface/diffusers
easyanimate model/pipeline review
- Vorherrschende Sprache
- Python
- Sterne
- 34.5k
- Forks
- 7.3k
- Ø Merge
- 3 T. 3 Std.
- Gemergte PRs (30 T.)
- 91
Beschreibung
# `easyanimate` model/pipeline review
Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`
Review performed against the repository review rules.
Duplicate search: checked GitHub Issues/PRs for `EasyAnimate`, affected class/function names, and the specific failure modes. No exact duplicates found. Related: https://github.com/huggingface/diffusers/issues/12646 reports another crash in the same inpaint repaint branch; https://github.com/huggingface/diffusers/pull/13347 only refactors transformer tests.
Local test note: `.venv` was used. Focused Python reproductions ran; full non-slow pytest collection is blocked in this `.venv` by `ModuleNotFoundError: torch._C._distributed_c10d`.
## Issue 1: `EasyAnimateControlPipeline.__call__` always passes an invalid `encode_prompt` kwarg
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_control.py#L796-L808
Problem:
`EasyAnimateControlPipeline.encode_prompt()` does not accept `text_encoder_index`, but `__call__` passes `text_encoder_index=0`. Any control pipeline call reaches this TypeError before denoising.
Impact:
`EasyAnimateControlPipeline` is effectively unusable through its public `__call__`.
Reproduction:
```python
from diffusers import EasyAnimateControlPipeline
pipe = object.__new__(EasyAnimateControlPipeline)
EasyAnimateControlPipeline.encode_prompt(pipe, prompt="x", text_encoder_index=0)
```
Relevant precedent:
Base and inpaint EasyAnimate call `encode_prompt` without this kwarg:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate.py#L653-L664
Suggested fix:
```python
# Remove the stale kwarg from EasyAnimateControlPipeline.__call__
negative_prompt_attention_mask=negative_prompt_attention_mask,
```
## Issue 2: `EasyAnimateControlPipeline` decodes through an undefined method
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_control.py#L985-L988
Problem:
The control pipeline calls `self.decode_latents(latents)`, but the class does not define `decode_latents`.
Impact:
After fixing Issue 1, any control run with `output_type != "latent"` will fail at decode time.
Reproduction:
```python
from diffusers import EasyAnimateControlPipeline
pipe = object.__new__(EasyAnimateControlPipeline)
print(hasattr(pipe, "decode_latents"))
pipe.decode_latents(None)
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate.py#L761-L764
Suggested fix:
```python
latents = 1 / self.vae.config.scaling_factor * latents
video = self.vae.decode(latents, return_dict=False)[0]
video = self.video_processor.postprocess_video(video=video, output_type=output_type)
```
## Issue 3: Control helper optional mask/reference paths are broken
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_control.py#L120-L149
Problem:
`get_video_to_video_latent()` calls `preprocess_image(..., size=sample_size)`, but the helper parameter is named `sample_size`. The `ref_image` branch also builds the wrong rank for the pipeline, which later expects `(B, C, F, H, W)`.
Impact:
Documented/public helper paths for control masks and reference images fail before pipeline execution.
Reproduction:
```python
from PIL import Image
from diffusers.pipelines.easyanimate.pipeline_easyanimate_control import get_video_to_video_latent
frame = Image.new("RGB", (8, 8), "white")
mask = Image.new("RGB", (8, 8), "black")
ref = Image.new("RGB", (8, 8), "blue")
for kwargs in ({"validation_video_mask": mask}, {"ref_image": ref}):
try:
get_video_to_video_latent([frame], 1, (8, 8), **kwargs)
except Exception as e:
print(type(e).__name__, e)
```
Relevant precedent:
The main video path already uses `sample_size=sample_size` at line 123.
Suggested fix:
```python
validation_video_mask = preprocess_image(validation_video_mask, sample_size=sample_size)[:1]
input_video_mask = torch.where(validation_video_mask < 240 / 255.0, 0.0, 255.0)
input_video_mask = input_video_mask.unsqueeze(0).unsqueeze(2).repeat(1, 1, input_video.shape[2], 1, 1)
ref_image = preprocess_image(ref_image, sample_size=sample_size)
ref_image = ref_image.unsqueeze(1).unsqueeze(0)
```
## Issue 4: Inpaint helper fails for multiple end frames
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_inpaint.py#L156-L164
Problem:
For `validation_image_end` as a list, the code slices with `len(end_video)`, which is the batch dimension (`1`), not the number of end frames.
Impact:
Multi-frame end conditioning raises a shape error and cannot prepare inputs.
Reproduction:
```python
from PIL import Image
from diffusers.pipelines.easyanimate.pipeline_easyanimate_inpaint import get_image_to_video_latent
start = Image.new("RGB", (8, 8), "white")
ends = [Image.new("RGB", (8, 8), "black"), Image.new("RGB", (8, 8), "blue")]
get_image_to_video_latent(start, ends, 4, (8, 8))
```
Relevant precedent:
The mask branch already uses `len(image_end)` on the next line.
Suggested fix:
```python
input_video[:, :, -len(image_end) :] = end_video
input_video_mask[:, :, -len(image_end) :] = 0
```
## Issue 5: Inpaint FlowMatch repaint branch has a malformed `torch.tensor` call
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_inpaint.py#L1194-L1202
Problem:
`torch.tensor([noise_timestep], noise)` passes `noise` as a second positional argument to `torch.tensor`, which is invalid. It also fails to pass `noise` to `scale_noise`.
Impact:
The repaint branch for `FlowMatchEulerDiscreteScheduler` crashes when `num_channels_transformer == num_channels_latents`.
Reproduction:
```python
import torch
from diffusers import FlowMatchEulerDiscreteScheduler
scheduler = FlowMatchEulerDiscreteScheduler()
scheduler.set_timesteps(2)
sample = torch.zeros(1, 4, 1, 2, 2)
noise = torch.ones_like(sample)
noise_timestep = scheduler.timesteps[1]
scheduler.scale_noise(sample, torch.tensor([noise_timestep], noise))
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/qwenimage/pipeline_qwenimage_inpaint.py#L1003-L1007
Suggested fix:
```python
init_latents_proper = self.scheduler.scale_noise(
init_latents_proper, torch.tensor([noise_timestep], device=device), noise
)
```
## Issue 6: EasyAnimate attention ignores Diffusers attention backends
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_easyanimate.py#L112-L182
Problem:
`EasyAnimateAttnProcessor2_0` calls `F.scaled_dot_product_attention` directly and does not define `_attention_backend` / `_parallel_config`. `model.set_attention_backend(...)` silently skips the processor.
Impact:
Users cannot select supported attention backends for EasyAnimate, and context-parallel/backend plumbing cannot affect this model.
Reproduction:
```python
from diffusers import EasyAnimateTransformer3DModel
model = EasyAnimateTransformer3DModel(
num_attention_heads=2,
attention_head_dim=16,
in_channels=4,
out_channels=4,
time_embed_dim=8,
text_embed_dim=16,
num_layers=1,
mmdit_layers=1,
patch_size=2,
)
processor = model.transformer_blocks[0].attn1.processor
print(hasattr(processor, "_attention_backend"))
model.set_attention_backend("native")
print(getattr(processor, "_attention_backend", None))
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_flux.py#L75-L125
Suggested fix:
```python
from ..attention_dispatch import dispatch_attention_fn
class EasyAnimateAttnProcessor2_0:
_attention_backend = None
_parallel_config = None
...
hidden_states = dispatch_attention_fn(
query,
key,
value,
attn_mask=attention_mask,
dropout_p=0.0,
is_causal=False,
backend=self._attention_backend,
parallel_config=self._parallel_config,
)
```
## Issue 7: Control and inpaint variants have no fast or slow pipeline coverage
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/easyanimate/test_easyanimate.py#L45-L46
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/easyanimate/test_easyanimate.py#L260-L296
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/docs/source/en/api/pipelines/easyanimate.md#L80-L88
Problem:
The test file only covers `EasyAnimatePipeline`; there are no fast tests or slow/integration tests for `EasyAnimateControlPipeline`, `EasyAnimateInpaintPipeline`, or their public helper functions. Docs also only autodoc the base pipeline.
Impact:
The concrete control/inpaint regressions above are not caught by CI, and public variants are less discoverable.
Reproduction:
```python
from pathlib import Path
tests = Path("tests/pipelines/easyanimate/test_easyanimate.py").read_text()
docs = Path("docs/source/en/api/pipelines/easyanimate.md").read_text()
print("EasyAnimateControlPipeline" in tests)
print("EasyAnimateInpaintPipeline" in tests)
print("[[autodoc]] EasyAnimateControlPipeline" in docs)
print("[[autodoc]] EasyAnimateInpaintPipeline" in docs)
```
Relevant precedent:
Existing base pipeline fast and slow tests are in the same file and can be extended with tiny control/inpaint fixtures.
Suggested fix:
Add focused fast tests for control and inpaint using tiny components, including `output_type="pt"` and `"latent"` paths plus helper utility tests. Add slow tests for official control and inpaint checkpoints, or explicitly mark/document why they cannot be run. Update the EasyAnimate docs to autodoc `EasyAnimateControlPipeline` and `EasyAnimateInpaintPipeline`.
Beitragsleitfaden
Rechercherichtung
Start with the affected EasyAnimate pipeline and transformer files named in the issue, then run the focused Python reproductions for the control, inpaint, and attention paths. Extend tests/pipelines/easyanimate/test_easyanimate.py and the EasyAnimate API docs as described, verifying the regressions and public variants; full pytest collection is currently blocked by the reported torch._C._distributed_c10d error.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- python, pytorch
- Bereich
- documentation, machine-learning, testing-qa
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Klar beschrieben
- Anfängerfreundlichkeit
- 45/100