huggingface / huggingface/diffusers
easyanimate model/pipeline review
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
# `easyanimate` model/pipeline review
Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`
Review performed against the repository review rules.
Duplicate search: checked GitHub Issues/PRs for `EasyAnimate`, affected class/function names, and the specific failure modes. No exact duplicates found. Related: https://github.com/huggingface/diffusers/issues/12646 reports another crash in the same inpaint repaint branch; https://github.com/huggingface/diffusers/pull/13347 only refactors transformer tests.
Local test note: `.venv` was used. Focused Python reproductions ran; full non-slow pytest collection is blocked in this `.venv` by `ModuleNotFoundError: torch._C._distributed_c10d`.
## Issue 1: `EasyAnimateControlPipeline.__call__` always passes an invalid `encode_prompt` kwarg
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_control.py#L796-L808
Problem:
`EasyAnimateControlPipeline.encode_prompt()` does not accept `text_encoder_index`, but `__call__` passes `text_encoder_index=0`. Any control pipeline call reaches this TypeError before denoising.
Impact:
`EasyAnimateControlPipeline` is effectively unusable through its public `__call__`.
Reproduction:
```python
from diffusers import EasyAnimateControlPipeline
pipe = object.__new__(EasyAnimateControlPipeline)
EasyAnimateControlPipeline.encode_prompt(pipe, prompt="x", text_encoder_index=0)
```
Relevant precedent:
Base and inpaint EasyAnimate call `encode_prompt` without this kwarg:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate.py#L653-L664
Suggested fix:
```python
# Remove the stale kwarg from EasyAnimateControlPipeline.__call__
negative_prompt_attention_mask=negative_prompt_attention_mask,
```
## Issue 2: `EasyAnimateControlPipeline` decodes through an undefined method
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_control.py#L985-L988
Problem:
The control pipeline calls `self.decode_latents(latents)`, but the class does not define `decode_latents`.
Impact:
After fixing Issue 1, any control run with `output_type != "latent"` will fail at decode time.
Reproduction:
```python
from diffusers import EasyAnimateControlPipeline
pipe = object.__new__(EasyAnimateControlPipeline)
print(hasattr(pipe, "decode_latents"))
pipe.decode_latents(None)
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate.py#L761-L764
Suggested fix:
```python
latents = 1 / self.vae.config.scaling_factor * latents
video = self.vae.decode(latents, return_dict=False)[0]
video = self.video_processor.postprocess_video(video=video, output_type=output_type)
```
## Issue 3: Control helper optional mask/reference paths are broken
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_control.py#L120-L149
Problem:
`get_video_to_video_latent()` calls `preprocess_image(..., size=sample_size)`, but the helper parameter is named `sample_size`. The `ref_image` branch also builds the wrong rank for the pipeline, which later expects `(B, C, F, H, W)`.
Impact:
Documented/public helper paths for control masks and reference images fail before pipeline execution.
Reproduction:
```python
from PIL import Image
from diffusers.pipelines.easyanimate.pipeline_easyanimate_control import get_video_to_video_latent
frame = Image.new("RGB", (8, 8), "white")
mask = Image.new("RGB", (8, 8), "black")
ref = Image.new("RGB", (8, 8), "blue")
for kwargs in ({"validation_video_mask": mask}, {"ref_image": ref}):
try:
get_video_to_video_latent([frame], 1, (8, 8), **kwargs)
except Exception as e:
print(type(e).__name__, e)
```
Relevant precedent:
The main video path already uses `sample_size=sample_size` at line 123.
Suggested fix:
```python
validation_video_mask = preprocess_image(validation_video_mask, sample_size=sample_size)[:1]
input_video_mask = torch.where(validation_video_mask < 240 / 255.0, 0.0, 255.0)
input_video_mask = input_video_mask.unsqueeze(0).unsqueeze(2).repeat(1, 1, input_video.shape[2], 1, 1)
ref_image = preprocess_image(ref_image, sample_size=sample_size)
ref_image = ref_image.unsqueeze(1).unsqueeze(0)
```
## Issue 4: Inpaint helper fails for multiple end frames
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_inpaint.py#L156-L164
Problem:
For `validation_image_end` as a list, the code slices with `len(end_video)`, which is the batch dimension (`1`), not the number of end frames.
Impact:
Multi-frame end conditioning raises a shape error and cannot prepare inputs.
Reproduction:
```python
from PIL import Image
from diffusers.pipelines.easyanimate.pipeline_easyanimate_inpaint import get_image_to_video_latent
start = Image.new("RGB", (8, 8), "white")
ends = [Image.new("RGB", (8, 8), "black"), Image.new("RGB", (8, 8), "blue")]
get_image_to_video_latent(start, ends, 4, (8, 8))
```
Relevant precedent:
The mask branch already uses `len(image_end)` on the next line.
Suggested fix:
```python
input_video[:, :, -len(image_end) :] = end_video
input_video_mask[:, :, -len(image_end) :] = 0
```
## Issue 5: Inpaint FlowMatch repaint branch has a malformed `torch.tensor` call
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/easyanimate/pipeline_easyanimate_inpaint.py#L1194-L1202
Problem:
`torch.tensor([noise_timestep], noise)` passes `noise` as a second positional argument to `torch.tensor`, which is invalid. It also fails to pass `noise` to `scale_noise`.
Impact:
The repaint branch for `FlowMatchEulerDiscreteScheduler` crashes when `num_channels_transformer == num_channels_latents`.
Reproduction:
```python
import torch
from diffusers import FlowMatchEulerDiscreteScheduler
scheduler = FlowMatchEulerDiscreteScheduler()
scheduler.set_timesteps(2)
sample = torch.zeros(1, 4, 1, 2, 2)
noise = torch.ones_like(sample)
noise_timestep = scheduler.timesteps[1]
scheduler.scale_noise(sample, torch.tensor([noise_timestep], noise))
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/qwenimage/pipeline_qwenimage_inpaint.py#L1003-L1007
Suggested fix:
```python
init_latents_proper = self.scheduler.scale_noise(
init_latents_proper, torch.tensor([noise_timestep], device=device), noise
)
```
## Issue 6: EasyAnimate attention ignores Diffusers attention backends
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_easyanimate.py#L112-L182
Problem:
`EasyAnimateAttnProcessor2_0` calls `F.scaled_dot_product_attention` directly and does not define `_attention_backend` / `_parallel_config`. `model.set_attention_backend(...)` silently skips the processor.
Impact:
Users cannot select supported attention backends for EasyAnimate, and context-parallel/backend plumbing cannot affect this model.
Reproduction:
```python
from diffusers import EasyAnimateTransformer3DModel
model = EasyAnimateTransformer3DModel(
num_attention_heads=2,
attention_head_dim=16,
in_channels=4,
out_channels=4,
time_embed_dim=8,
text_embed_dim=16,
num_layers=1,
mmdit_layers=1,
patch_size=2,
)
processor = model.transformer_blocks[0].attn1.processor
print(hasattr(processor, "_attention_backend"))
model.set_attention_backend("native")
print(getattr(processor, "_attention_backend", None))
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_flux.py#L75-L125
Suggested fix:
```python
from ..attention_dispatch import dispatch_attention_fn
class EasyAnimateAttnProcessor2_0:
_attention_backend = None
_parallel_config = None
...
hidden_states = dispatch_attention_fn(
query,
key,
value,
attn_mask=attention_mask,
dropout_p=0.0,
is_causal=False,
backend=self._attention_backend,
parallel_config=self._parallel_config,
)
```
## Issue 7: Control and inpaint variants have no fast or slow pipeline coverage
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/easyanimate/test_easyanimate.py#L45-L46
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/easyanimate/test_easyanimate.py#L260-L296
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/docs/source/en/api/pipelines/easyanimate.md#L80-L88
Problem:
The test file only covers `EasyAnimatePipeline`; there are no fast tests or slow/integration tests for `EasyAnimateControlPipeline`, `EasyAnimateInpaintPipeline`, or their public helper functions. Docs also only autodoc the base pipeline.
Impact:
The concrete control/inpaint regressions above are not caught by CI, and public variants are less discoverable.
Reproduction:
```python
from pathlib import Path
tests = Path("tests/pipelines/easyanimate/test_easyanimate.py").read_text()
docs = Path("docs/source/en/api/pipelines/easyanimate.md").read_text()
print("EasyAnimateControlPipeline" in tests)
print("EasyAnimateInpaintPipeline" in tests)
print("[[autodoc]] EasyAnimateControlPipeline" in docs)
print("[[autodoc]] EasyAnimateInpaintPipeline" in docs)
```
Relevant precedent:
Existing base pipeline fast and slow tests are in the same file and can be extended with tiny control/inpaint fixtures.
Suggested fix:
Add focused fast tests for control and inpaint using tiny components, including `output_type="pt"` and `"latent"` paths plus helper utility tests. Add slow tests for official control and inpaint checkpoints, or explicitly mark/document why they cannot be run. Update the EasyAnimate docs to autodoc `EasyAnimateControlPipeline` and `EasyAnimateInpaintPipeline`.
Contributor guide
Research direction
Start with the affected EasyAnimate pipeline and transformer files named in the issue, then run the focused Python reproductions for the control, inpaint, and attention paths. Extend tests/pipelines/easyanimate/test_easyanimate.py and the EasyAnimate API docs as described, verifying the regressions and public variants; full pytest collection is currently blocked by the reported torch._C._distributed_c10d error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- documentation, machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100