huggingface / huggingface/diffusers

t2i_adapter model/pipeline review

Abierto
#13,626 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
34.5k
Forks
7.3k
Merge medio
3 d 3 h
PR fusionados (30 d)
91

Descripción

# `t2i_adapter` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

Reviewed: target model/pipeline files, public exports/lazy imports, serialization/loading, dtype/device/offload paths, related SD/SDXL precedents, fast/slow tests, docs, and examples. Public imports and lazy-loading registration look correct.

Duplicate searches run with `gh search issues/prs` for `t2i_adapter`, affected class names, `MultiAdapter`, `adapter_conditioning_scale`, `iteration over a 0-d tensor`, SDXL list adapters, latent output, PathLike save/load, docs scheduler typo, and slow coverage.

## Issue 1: `MultiAdapter` still breaks on the pipeline default scale

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L88-L94
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L884-L885
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L1166-L1167

Problem:
Both pipelines pass the default `adapter_conditioning_scale=1.0` to `MultiAdapter.forward`. `MultiAdapter.forward` converts that float to a scalar tensor and then iterates it, raising `TypeError: iteration over a 0-d tensor`. It also silently truncates when a scale list has the wrong length.

Duplicate check:
This exact default-scale failure was reported in closed issue https://github.com/huggingface/diffusers/issues/6274 and still reproduces on this commit, so this is not a new finding.

Impact:
A documented/default multi-adapter call fails unless users know to pass a list. Wrong-length scale lists can silently skip adapters.

Reproduction:
```python
import torch
from diffusers import MultiAdapter, T2IAdapter

multi = MultiAdapter([
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
])
xs = [torch.randn(1, 3, 8, 8), torch.randn(1, 3, 8, 8)]

try:
multi(xs, 1.0)
except Exception as e:
print(type(e).__name__, str(e))

print("short list accepted:", multi(xs, [1.0])[0].shape)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/community/pipeline_stable_diffusion_xl_controlnet_adapter.py#L1089-L1090

Suggested fix:
```python
if adapter_weights is None:
adapter_weights = [1 / self.num_adapter] * self.num_adapter
elif isinstance(adapter_weights, (float, int)):
adapter_weights = [float(adapter_weights)] * self.num_adapter
elif len(adapter_weights) != self.num_adapter:
raise ValueError(
f"`adapter_weights` must have length {self.num_adapter}, got {len(adapter_weights)}."
)

if len(xs) != self.num_adapter:
raise ValueError(f"`xs` must have length {self.num_adapter}, got {len(xs)}.")
```

## Issue 2: SDXL adapter pipeline does not accept `list[T2IAdapter]` despite its public signature

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L273-L290

Problem:
`StableDiffusionXLAdapterPipeline.__init__` documents and types `adapter` as `T2IAdapter | MultiAdapter | list[T2IAdapter]`, but registers the raw list. `register_modules` then fails because a Python list has no `__module__`.

Impact:
SDXL is inconsistent with the SD adapter pipeline and breaks a documented constructor form.

Reproduction:
```python
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter

try:
StableDiffusionXLAdapterPipeline(
vae=None, text_encoder=None, text_encoder_2=None,
tokenizer=None, tokenizer_2=None, unet=None, scheduler=None,
adapter=[
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
],
)
except Exception as e:
print(type(e).__name__, str(e))
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L260-L261

Suggested fix:
```python
if isinstance(adapter, (list, tuple)):
adapter = MultiAdapter(adapter)

self.register_modules(
vae=vae,
text_encoder=text_encoder,
text_encoder_2=text_encoder_2,
tokenizer=tokenizer,
tokenizer_2=tokenizer_2,
unet=unet,
adapter=adapter,
scheduler=scheduler,
feature_extractor=feature_extractor,
image_encoder=image_encoder,
)
```

## Issue 3: SDXL latent output returns before cleanup and ignores `return_dict=False`

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L1278-L1298

Problem:
For `output_type="latent"`, `StableDiffusionXLAdapterPipeline.__call__` returns immediately, before `maybe_free_model_hooks()` and before the `return_dict` handling.

Impact:
Model offload hooks are not released on latent output, and `return_dict=False` still returns `StableDiffusionXLPipelineOutput`.

Reproduction:
```python
import types
import torch
from diffusers import AutoencoderKL, EulerDiscreteScheduler, StableDiffusionXLAdapterPipeline, T2IAdapter, UNet2DConditionModel

unet = UNet2DConditionModel(
block_out_channels=(32, 64), layers_per_block=1, sample_size=32,
in_channels=4, out_channels=4,
down_block_types=("DownBlock2D", "CrossAttnDownBlock2D"),
up_block_types=("CrossAttnUpBlock2D", "UpBlock2D"),
attention_head_dim=(2, 4), use_linear_projection=True,
addition_embed_type="text_time", addition_time_embed_dim=8,
transformer_layers_per_block=(1, 1),
projection_class_embeddings_input_dim=80, cross_attention_dim=64,
)
vae = AutoencoderKL(
block_out_channels=[32, 64], in_channels=3, out_channels=3,
down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"],
up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D"], latent_channels=4,
)
pipe = StableDiffusionXLAdapterPipeline(
vae=vae, text_encoder=None, text_encoder_2=None, tokenizer=None, tokenizer_2=None,
unet=unet,
adapter=T2IAdapter(in_channels=3, channels=[32, 64], num_res_blocks=1, downscale_factor=4, adapter_type="full_adapter_xl"),
scheduler=EulerDiscreteScheduler(),
)
pipe.set_progress_bar_config(disable=True)
pipe.freed = False
pipe.maybe_free_model_hooks = types.MethodType(lambda self: setattr(self, "freed", True), pipe)

out = pipe(
prompt_embeds=torch.zeros(1, 2, 64),
negative_prompt_embeds=torch.zeros(1, 2, 64),
pooled_prompt_embeds=torch.zeros(1, 32),
negative_pooled_prompt_embeds=torch.zeros(1, 32),
image=torch.zeros(1, 3, 64, 64),
num_inference_steps=1,
guidance_scale=1.0,
output_type="latent",
return_dict=False,
)
print(type(out).__name__, pipe.freed)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py#L1287-L1300

Suggested fix:
```python
else:
image = latents

if not output_type == "latent":
image = self.image_processor.postprocess(image, output_type=output_type)

self.maybe_free_model_hooks()

if not return_dict:
return (image,)

return StableDiffusionXLPipelineOutput(images=image)
```

## Issue 4: `MultiAdapter.save_pretrained` and `from_pretrained` reject `PathLike`

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L130-L145
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L196-L204

Problem:
The signatures accept `str | os.PathLike`, but the implementation concatenates paths with `+ f"_{idx}"`, which fails for `pathlib.Path`.

Impact:
Serialization/loading works with strings but fails with standard path objects.

Reproduction:
```python
from pathlib import Path
import tempfile
from diffusers import MultiAdapter, T2IAdapter

multi = MultiAdapter([
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
])

with tempfile.TemporaryDirectory() as d:
try:
multi.save_pretrained(Path(d) / "adapter")
except Exception as e:
print("save:", type(e).__name__, str(e))

with tempfile.TemporaryDirectory() as d:
path = Path(d) / "adapter"
multi.save_pretrained(str(path))
try:
MultiAdapter.from_pretrained(path)
except Exception as e:
print("load:", type(e).__name__, str(e))
```

Relevant precedent:
`T2IAdapter` inherits the normal `ModelMixin` path handling; this custom override should preserve the same public contract.

Suggested fix:
```python
save_directory = os.fspath(save_directory)
...
model_path_to_save = f"{save_directory}_{idx}"

pretrained_model_path = os.fspath(pretrained_model_path)
...
model_path_to_load = f"{pretrained_model_path}_{idx}"
```

## Issue 5: SD adapter has dead LoRA/textual-inversion hooks because it does not inherit the loader mixins

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L25
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L191
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L354-L372

Problem:
`StableDiffusionAdapterPipeline` imports `StableDiffusionLoraLoaderMixin` and `TextualInversionLoaderMixin`, and `encode_prompt` checks for them, but the class does not inherit either mixin.

Impact:
`StableDiffusionAdapterPipeline` cannot load LoRA or textual inversion, unlike `StableDiffusionPipeline` and `StableDiffusionXLAdapterPipeline`.

Reproduction:
```python
from diffusers import StableDiffusionAdapterPipeline, StableDiffusionPipeline, StableDiffusionXLAdapterPipeline

for cls in [StableDiffusionPipeline, StableDiffusionAdapterPipeline, StableDiffusionXLAdapterPipeline]:
print(cls.__name__, hasattr(cls, "load_lora_weights"), hasattr(cls, "load_textual_inversion"))
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L154-L160
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L213-L220

Suggested fix:
```python
class StableDiffusionAdapterPipeline(
DiffusionPipeline,
StableDiffusionMixin,
TextualInversionLoaderMixin,
StableDiffusionLoraLoaderMixin,
FromSingleFileMixin,
):
...
```

## Issue 6: T2I-Adapter docs import a nonexistent scheduler class

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/t2i_adapter/README_sdxl.md#L97-L110
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/docs/source/en/training/t2i_adapters.md#L191-L200

Problem:
The inference snippets import `EulerAncestralDiscreteSchedulerTest`, which is not exported. The training docs also assign from `pipe.scheduler.config` while the variable is named `pipeline`.

Impact:
Users following the example hit an immediate import/name error.

Reproduction:
```python
try:
from diffusers import EulerAncestralDiscreteSchedulerTest
except Exception as e:
print(type(e).__name__, str(e))
```

Relevant precedent:
Use the public scheduler class exported by diffusers.

Suggested fix:
```python
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, EulerAncestralDiscreteScheduler

...
pipeline.scheduler = EulerAncestralDiscreteScheduler.from_config(pipeline.scheduler.config)
```

## Issue 7: SDXL adapter lacks a plain slow golden test in its pipeline test file

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py#L52
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/stable_diffusion_adapter/test_stable_diffusion_adapter.py#L607-L609

Problem:
Fast SDXL adapter tests exist, and there are SDXL adapter slow paths in single-file and LoRA integration tests, but `tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py` has no plain slow golden inference test for the default SDXL adapter pipeline.

Impact:
Core SDXL adapter behavior can regress without a direct slow pipeline fixture. The `output_type="latent"` return bug and constructor/list handling are not covered by existing slow SDXL adapter tests.

Reproduction:
```python
from pathlib import Path

sdxl_test = Path("tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py").read_text()
sd_test = Path("tests/pipelines/stable_diffusion_adapter/test_stable_diffusion_adapter.py").read_text()

print("@slow in SDXL adapter pipeline test:", "@slow" in sdxl_test)
print("@slow in SD adapter pipeline test:", "@slow" in sd_test)
```

Relevant precedent:
The SD adapter pipeline has a dedicated slow class with real adapter checkpoints and expected arrays.

Suggested fix:
Add a `@slow` SDXL adapter pipeline regression test in `tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py`, using an `hf-internal-testing` image and a stable expected array under `datasets/diffusers/test-arrays`, covering at least normal inference and `output_type="latent", return_dict=False`.

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Start with the affected adapter.py and Stable Diffusion adapter pipeline files, then run the six reproductions in the issue to confirm each reported failure. Review the referenced SD/SDXL pipeline precedents and the README_sdxl.md and training/t2i_adapters.md examples. Done means the listed adapter defaults, constructor, latent output, PathLike serialization, loader mixins, and documentation examples behave as documented.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python, pytorch
Área
documentation, machine-learning
Tipo de issue
Error
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.