huggingface / huggingface/diffusers

hunyuan_image model/pipeline review

Offen
#13,605 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

# `hunyuan_image` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

Duplicate-search status: searched `hunyuan_image`, `HunyuanImagePipeline`, `AutoencoderKLHunyuanImage`, `AutoencoderKLHunyuanImageRefiner`, plus targeted tiling/shortcut/shape terms. Broad matches were the original integration PR https://github.com/huggingface/diffusers/pull/12333 and HunyuanImage 3.0 request https://github.com/huggingface/diffusers/issues/12412; I did not find duplicates for the issues below. A second `gh search` batch hit GitHub API rate limits, so I completed targeted duplicate checks with web search.

## Issue 1: Base VAE default scaling factor breaks pipeline decode

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_hunyuanimage.py#L423-L465
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_image/pipeline_hunyuanimage.py#L855-L860

Problem:
`AutoencoderKLHunyuanImage` defaults `scaling_factor` to `None`, but `HunyuanImagePipeline` always divides latents by `self.vae.config.scaling_factor` before decoding.

Impact:
A pipeline assembled from default tiny components can denoise successfully and then fail at decode with `TypeError`. The refiner VAE already has a numeric default.

Reproduction:
```python
import torch
from diffusers import AutoencoderKLHunyuanImage, FlowMatchEulerDiscreteScheduler, HunyuanImagePipeline, HunyuanImageTransformer2DModel

transformer = HunyuanImageTransformer2DModel(
in_channels=4, out_channels=4, num_attention_heads=1, attention_head_dim=4,
num_layers=0, num_single_layers=0, num_refiner_layers=0,
patch_size=(1, 1), text_embed_dim=4, text_embed_2_dim=4, rope_axes_dim=(2, 2),
)
vae = AutoencoderKLHunyuanImage(
in_channels=3, out_channels=3, latent_channels=4, block_out_channels=(32,),
layers_per_block=1, spatial_compression_ratio=8, sample_size=32,
)
pipe = HunyuanImagePipeline(
scheduler=FlowMatchEulerDiscreteScheduler(), vae=vae, text_encoder=None, tokenizer=None,
text_encoder_2=None, tokenizer_2=None, transformer=transformer, guider=None, ocr_guider=None,
)
pipe(
prompt=None,
prompt_embeds=torch.randn(1, 1, 4),
prompt_embeds_mask=torch.ones(1, 1, dtype=torch.long),
prompt_embeds_2=torch.zeros(1, 1, 4),
prompt_embeds_mask_2=torch.zeros(1, 1, dtype=torch.long),
height=16, width=16, num_inference_steps=1, output_type="pt",
)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_hunyuanimage_refiner.py#L604-L617

Suggested fix:
```python
scaling_factor: float = 0.476986
```

## Issue 2: `HunyuanImagePipelineOutput` is not exported from the pipeline package

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_image/__init__.py#L25-L37
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_image/pipeline_output.py#L10-L20

Problem:
The output class exists and is documented, but `diffusers.pipelines.hunyuan_image` only lazy-exports the two pipelines.

Impact:
Users cannot import the output type from the package namespace, unlike similar pipeline families.

Reproduction:
```python
from diffusers.pipelines.hunyuan_image import HunyuanImagePipelineOutput
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/qwenimage/__init__.py#L15-L43

Suggested fix:
```python
_import_structure["pipeline_output"] = ["HunyuanImagePipelineOutput"]
...
from .pipeline_output import HunyuanImagePipelineOutput
```

## Issue 3: Base VAE tiled encode crashes on 4D image tensors

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_hunyuanimage.py#L582-L620

Problem:
`AutoencoderKLHunyuanImage` is a 2D image VAE, but `tiled_encode()` unpacks `x.shape` as if it were 5D and slices tiles with an extra temporal dimension.

Impact:
`vae.enable_tiling(); vae.encode(image)` fails for image tensors large enough to trigger tiling. Existing tests cover tiled decode through the pipeline, but not tiled encode.

Reproduction:
```python
import torch
from diffusers import AutoencoderKLHunyuanImage

vae = AutoencoderKLHunyuanImage(
in_channels=3, out_channels=3, latent_channels=4, block_out_channels=(32,),
layers_per_block=1, spatial_compression_ratio=1, sample_size=8,
)
vae.enable_tiling(tile_sample_min_size=4)
vae.encode(torch.randn(1, 3, 8, 8))
```

Relevant precedent:
The same file's `tiled_decode()` correctly treats base VAE latents as 4D:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_hunyuanimage.py#L622-L664

Suggested fix:
```python
_, _, height, width = x.shape
...
tile = x[:, :, i : i + self.tile_sample_min_size, j : j + self.tile_sample_min_size]
...
result_row.append(tile[:, :, :row_limit, :row_limit])
```

## Issue 4: Refiner VAE tiling mixes sample and latent units

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_hunyuanimage_refiner.py#L793-L841
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_hunyuanimage_refiner.py#L843-L895

Problem:
`tiled_encode()` iterates over sample-space height/width using latent-space overlap values. `tiled_decode()` crops decoded sample tiles using latent-space row limits.

Impact:
Tiled encode can generate invalid edge tiles, and tiled decode returns the wrong spatial size.

Reproduction:
```python
import torch
from diffusers import AutoencoderKLHunyuanImageRefiner

vae = AutoencoderKLHunyuanImageRefiner(
in_channels=3, out_channels=3, latent_channels=4, block_out_channels=(8, 8),
layers_per_block=0, spatial_compression_ratio=2, temporal_compression_ratio=1,
)
z = torch.randn(1, 4, 1, 8, 8)
plain = vae.decode(z).sample.shape
vae.enable_tiling(tile_sample_min_height=8, tile_sample_min_width=8, tile_overlap_factor=0.25)
tiled = vae.decode(z).sample.shape
print(plain, tiled) # plain is [1, 3, 1, 16, 16], tiled is [1, 3, 1, 9, 9]
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_hunyuanvideo15.py#L829-L899

Suggested fix:
Use sample-space stride/overlap for encode iteration, latent-space crop sizes for encoded tiles, latent-space stride for decode iteration, and sample-space crop sizes for decoded tiles. The existing `tile_sample_stride_height` / `tile_sample_stride_width` fields should either be used or removed.

## Issue 5: Base VAE shortcut projection is applied to the wrong tensor

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_hunyuanimage.py#L61-L77

Problem:
When `in_channels != out_channels`, `HunyuanImageResnetBlock.forward()` applies `conv_shortcut` to `x` after `conv2`, not to the saved residual.

Impact:
Non-default serialized configs with `downsample_match_channel=False` or `upsample_match_channel=False` crash when a resnet block changes channels.

Reproduction:
```python
import torch
from diffusers import AutoencoderKLHunyuanImage

vae = AutoencoderKLHunyuanImage(
in_channels=3, out_channels=3, latent_channels=4, block_out_channels=(32, 64),
layers_per_block=1, spatial_compression_ratio=2, sample_size=16,
downsample_match_channel=False,
)
vae.encode(torch.randn(1, 3, 16, 16))
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_hunyuanimage_refiner.py#L253-L267

Suggested fix:
```python
if self.conv_shortcut is not None:
residual = self.conv_shortcut(residual)
return x + residual
```

## Issue 6: Transformer silently truncates latent sizes not divisible by `patch_size`

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_hunyuanimage.py#L766-L885
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_image/pipeline_hunyuanimage.py#L406-L486
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/hunyuan_image/pipeline_hunyuanimage_refiner.py#L295-L375

Problem:
The transformer computes `post_patch_sizes = d // p` and unpatchifies to `post_patch * patch`, silently dropping remainder pixels. The pipelines warn about divisibility but do not round latent sizes to the transformer patch grid.

Impact:
Custom configs with `patch_size > 1` can produce a model output smaller than the input latents, leading to scheduler shape errors or silent shape loss when the model is used directly.

Reproduction:
```python
import torch
from diffusers import HunyuanImageTransformer2DModel

model = HunyuanImageTransformer2DModel(
in_channels=4, out_channels=4, num_attention_heads=1, attention_head_dim=4,
num_layers=0, num_single_layers=0, num_refiner_layers=0,
patch_size=(2, 2), text_embed_dim=4, rope_axes_dim=(2, 2),
)
out = model(
hidden_states=torch.randn(1, 4, 3, 3),
timestep=torch.ones(1),
encoder_hidden_states=torch.randn(1, 1, 4),
encoder_attention_mask=torch.ones(1, 1, dtype=torch.long),
).sample
print(out.shape) # torch.Size([1, 4, 2, 2])
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/qwenimage/pipeline_qwenimage.py#L346-L348

Suggested fix:
```python
if any(size % patch != 0 for size, patch in zip(sizes, self.config.patch_size)):
raise ValueError(f"`hidden_states` spatial/temporal sizes {sizes} must be divisible by patch_size {self.config.patch_size}.")
```

## Issue 7: Base VAE has checkpointing code but disables the public capability flag

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_hunyuanimage.py#L296-L308
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_hunyuanimage.py#L393-L403
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_hunyuanimage.py#L420-L420

Problem:
The encoder and decoder contain `self._gradient_checkpointing_func(...)` branches, but `AutoencoderKLHunyuanImage._supports_gradient_checkpointing` is `False`.

Impact:
Training or fine-tuning code cannot enable gradient checkpointing on the base VAE even though the implementation paths exist.

Reproduction:
```python
from diffusers import AutoencoderKLHunyuanImage

vae = AutoencoderKLHunyuanImage(
in_channels=3, out_channels=3, latent_channels=4, block_out_channels=(32,),
layers_per_block=1, spatial_compression_ratio=1, sample_size=8,
)
vae.enable_gradient_checkpointing()
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/autoencoders/autoencoder_kl_hunyuanimage_refiner.py#L602-L602

Suggested fix:
```python
_supports_gradient_checkpointing = True
```

## Issue 8: Test coverage is incomplete, including missing slow tests

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/hunyuan_image_21/test_hunyuanimage.py#L43-L48

Problem:
There is one fast pipeline test class for `HunyuanImagePipeline`. I found no model fast tests for `HunyuanImageTransformer2DModel`, `AutoencoderKLHunyuanImage`, or `AutoencoderKLHunyuanImageRefiner`, no fast tests for `HunyuanImageRefinerPipeline`, and no `@slow` tests for this family.

Impact:
The broken VAE encode tiling, refiner tiling, refiner pipeline paths, model patch-size behavior, and default scaling-factor failure are not covered.

Reproduction:
```python
from pathlib import Path

files = list(Path("tests").rglob("*.py"))
hunyuan_files = [p for p in files if "hunyuan_image" in str(p).lower() or "hunyuanimage" in p.read_text(encoding="utf-8", errors="ignore")]
print([str(p) for p in hunyuan_files])
print("has_refiner_pipeline_test", any("HunyuanImageRefinerPipeline" in p.read_text(encoding="utf-8", errors="ignore") for p in hunyuan_files))
print("has_slow_test", any("@slow" in p.read_text(encoding="utf-8", errors="ignore") for p in hunyuan_files))
```

Relevant precedent:
Other families keep slow pipeline tests alongside fast tests, for example:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/wan/test_wan.py#L185-L185

Suggested fix:
Add focused model tests under `tests/models/` for the transformer and both VAEs, add fast refiner pipeline tests with tiny components, and add at least one slow test for the base and refiner pretrained checkpoints. I attempted the existing fast Hunyuan test file locally with `.venv`, but collection failed before running tests because this environment's PyTorch build lacks `torch._C._distributed_c10d`.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne damit, die Überprüfung in die im Issue genannten betroffenen VAE-, Pipeline-, Transformer- und Testpfade aufzuteilen, und führe dann die bereitgestellten Reproduktionen gegen den Commit 0f1abc4ae8b0eb2a3b40e82a310507281144c423 aus. Vergleiche die Implementierungen von Base und Refiner sowie die vorhandenen Tests; als abgeschlossen gilt die Aufgabe, wenn eine Regressionstestabdeckung für die gemeldeten Fehler vorhanden ist und bestätigt wurde, dass die Tests der Hunyuan Image pipeline bestehen.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, pytorch
Bereich
machine-learning, testing-qa
Issue-Typ
Bug
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Ruhig
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
30/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.