huggingface / huggingface/diffusers

t2i_adapter model/pipeline review

Aperta
#13,626 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

# `t2i_adapter` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

Reviewed: target model/pipeline files, public exports/lazy imports, serialization/loading, dtype/device/offload paths, related SD/SDXL precedents, fast/slow tests, docs, and examples. Public imports and lazy-loading registration look correct.

Duplicate searches run with `gh search issues/prs` for `t2i_adapter`, affected class names, `MultiAdapter`, `adapter_conditioning_scale`, `iteration over a 0-d tensor`, SDXL list adapters, latent output, PathLike save/load, docs scheduler typo, and slow coverage.

## Issue 1: `MultiAdapter` still breaks on the pipeline default scale

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L88-L94
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L884-L885
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L1166-L1167

Problem:
Both pipelines pass the default `adapter_conditioning_scale=1.0` to `MultiAdapter.forward`. `MultiAdapter.forward` converts that float to a scalar tensor and then iterates it, raising `TypeError: iteration over a 0-d tensor`. It also silently truncates when a scale list has the wrong length.

Duplicate check:
This exact default-scale failure was reported in closed issue https://github.com/huggingface/diffusers/issues/6274 and still reproduces on this commit, so this is not a new finding.

Impact:
A documented/default multi-adapter call fails unless users know to pass a list. Wrong-length scale lists can silently skip adapters.

Reproduction:
```python
import torch
from diffusers import MultiAdapter, T2IAdapter

multi = MultiAdapter([
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
])
xs = [torch.randn(1, 3, 8, 8), torch.randn(1, 3, 8, 8)]

try:
multi(xs, 1.0)
except Exception as e:
print(type(e).__name__, str(e))

print("short list accepted:", multi(xs, [1.0])[0].shape)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/community/pipeline_stable_diffusion_xl_controlnet_adapter.py#L1089-L1090

Suggested fix:
```python
if adapter_weights is None:
adapter_weights = [1 / self.num_adapter] * self.num_adapter
elif isinstance(adapter_weights, (float, int)):
adapter_weights = [float(adapter_weights)] * self.num_adapter
elif len(adapter_weights) != self.num_adapter:
raise ValueError(
f"`adapter_weights` must have length {self.num_adapter}, got {len(adapter_weights)}."
)

if len(xs) != self.num_adapter:
raise ValueError(f"`xs` must have length {self.num_adapter}, got {len(xs)}.")
```

## Issue 2: SDXL adapter pipeline does not accept `list[T2IAdapter]` despite its public signature

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L273-L290

Problem:
`StableDiffusionXLAdapterPipeline.__init__` documents and types `adapter` as `T2IAdapter | MultiAdapter | list[T2IAdapter]`, but registers the raw list. `register_modules` then fails because a Python list has no `__module__`.

Impact:
SDXL is inconsistent with the SD adapter pipeline and breaks a documented constructor form.

Reproduction:
```python
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter

try:
StableDiffusionXLAdapterPipeline(
vae=None, text_encoder=None, text_encoder_2=None,
tokenizer=None, tokenizer_2=None, unet=None, scheduler=None,
adapter=[
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
],
)
except Exception as e:
print(type(e).__name__, str(e))
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L260-L261

Suggested fix:
```python
if isinstance(adapter, (list, tuple)):
adapter = MultiAdapter(adapter)

self.register_modules(
vae=vae,
text_encoder=text_encoder,
text_encoder_2=text_encoder_2,
tokenizer=tokenizer,
tokenizer_2=tokenizer_2,
unet=unet,
adapter=adapter,
scheduler=scheduler,
feature_extractor=feature_extractor,
image_encoder=image_encoder,
)
```

## Issue 3: SDXL latent output returns before cleanup and ignores `return_dict=False`

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L1278-L1298

Problem:
For `output_type="latent"`, `StableDiffusionXLAdapterPipeline.__call__` returns immediately, before `maybe_free_model_hooks()` and before the `return_dict` handling.

Impact:
Model offload hooks are not released on latent output, and `return_dict=False` still returns `StableDiffusionXLPipelineOutput`.

Reproduction:
```python
import types
import torch
from diffusers import AutoencoderKL, EulerDiscreteScheduler, StableDiffusionXLAdapterPipeline, T2IAdapter, UNet2DConditionModel

unet = UNet2DConditionModel(
block_out_channels=(32, 64), layers_per_block=1, sample_size=32,
in_channels=4, out_channels=4,
down_block_types=("DownBlock2D", "CrossAttnDownBlock2D"),
up_block_types=("CrossAttnUpBlock2D", "UpBlock2D"),
attention_head_dim=(2, 4), use_linear_projection=True,
addition_embed_type="text_time", addition_time_embed_dim=8,
transformer_layers_per_block=(1, 1),
projection_class_embeddings_input_dim=80, cross_attention_dim=64,
)
vae = AutoencoderKL(
block_out_channels=[32, 64], in_channels=3, out_channels=3,
down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"],
up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D"], latent_channels=4,
)
pipe = StableDiffusionXLAdapterPipeline(
vae=vae, text_encoder=None, text_encoder_2=None, tokenizer=None, tokenizer_2=None,
unet=unet,
adapter=T2IAdapter(in_channels=3, channels=[32, 64], num_res_blocks=1, downscale_factor=4, adapter_type="full_adapter_xl"),
scheduler=EulerDiscreteScheduler(),
)
pipe.set_progress_bar_config(disable=True)
pipe.freed = False
pipe.maybe_free_model_hooks = types.MethodType(lambda self: setattr(self, "freed", True), pipe)

out = pipe(
prompt_embeds=torch.zeros(1, 2, 64),
negative_prompt_embeds=torch.zeros(1, 2, 64),
pooled_prompt_embeds=torch.zeros(1, 32),
negative_pooled_prompt_embeds=torch.zeros(1, 32),
image=torch.zeros(1, 3, 64, 64),
num_inference_steps=1,
guidance_scale=1.0,
output_type="latent",
return_dict=False,
)
print(type(out).__name__, pipe.freed)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py#L1287-L1300

Suggested fix:
```python
else:
image = latents

if not output_type == "latent":
image = self.image_processor.postprocess(image, output_type=output_type)

self.maybe_free_model_hooks()

if not return_dict:
return (image,)

return StableDiffusionXLPipelineOutput(images=image)
```

## Issue 4: `MultiAdapter.save_pretrained` and `from_pretrained` reject `PathLike`

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L130-L145
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L196-L204

Problem:
The signatures accept `str | os.PathLike`, but the implementation concatenates paths with `+ f"_{idx}"`, which fails for `pathlib.Path`.

Impact:
Serialization/loading works with strings but fails with standard path objects.

Reproduction:
```python
from pathlib import Path
import tempfile
from diffusers import MultiAdapter, T2IAdapter

multi = MultiAdapter([
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
])

with tempfile.TemporaryDirectory() as d:
try:
multi.save_pretrained(Path(d) / "adapter")
except Exception as e:
print("save:", type(e).__name__, str(e))

with tempfile.TemporaryDirectory() as d:
path = Path(d) / "adapter"
multi.save_pretrained(str(path))
try:
MultiAdapter.from_pretrained(path)
except Exception as e:
print("load:", type(e).__name__, str(e))
```

Relevant precedent:
`T2IAdapter` inherits the normal `ModelMixin` path handling; this custom override should preserve the same public contract.

Suggested fix:
```python
save_directory = os.fspath(save_directory)
...
model_path_to_save = f"{save_directory}_{idx}"

pretrained_model_path = os.fspath(pretrained_model_path)
...
model_path_to_load = f"{pretrained_model_path}_{idx}"
```

## Issue 5: SD adapter has dead LoRA/textual-inversion hooks because it does not inherit the loader mixins

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L25
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L191
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L354-L372

Problem:
`StableDiffusionAdapterPipeline` imports `StableDiffusionLoraLoaderMixin` and `TextualInversionLoaderMixin`, and `encode_prompt` checks for them, but the class does not inherit either mixin.

Impact:
`StableDiffusionAdapterPipeline` cannot load LoRA or textual inversion, unlike `StableDiffusionPipeline` and `StableDiffusionXLAdapterPipeline`.

Reproduction:
```python
from diffusers import StableDiffusionAdapterPipeline, StableDiffusionPipeline, StableDiffusionXLAdapterPipeline

for cls in [StableDiffusionPipeline, StableDiffusionAdapterPipeline, StableDiffusionXLAdapterPipeline]:
print(cls.__name__, hasattr(cls, "load_lora_weights"), hasattr(cls, "load_textual_inversion"))
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L154-L160
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L213-L220

Suggested fix:
```python
class StableDiffusionAdapterPipeline(
DiffusionPipeline,
StableDiffusionMixin,
TextualInversionLoaderMixin,
StableDiffusionLoraLoaderMixin,
FromSingleFileMixin,
):
...
```

## Issue 6: T2I-Adapter docs import a nonexistent scheduler class

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/t2i_adapter/README_sdxl.md#L97-L110
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/docs/source/en/training/t2i_adapters.md#L191-L200

Problem:
The inference snippets import `EulerAncestralDiscreteSchedulerTest`, which is not exported. The training docs also assign from `pipe.scheduler.config` while the variable is named `pipeline`.

Impact:
Users following the example hit an immediate import/name error.

Reproduction:
```python
try:
from diffusers import EulerAncestralDiscreteSchedulerTest
except Exception as e:
print(type(e).__name__, str(e))
```

Relevant precedent:
Use the public scheduler class exported by diffusers.

Suggested fix:
```python
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, EulerAncestralDiscreteScheduler

...
pipeline.scheduler = EulerAncestralDiscreteScheduler.from_config(pipeline.scheduler.config)
```

## Issue 7: SDXL adapter lacks a plain slow golden test in its pipeline test file

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py#L52
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/stable_diffusion_adapter/test_stable_diffusion_adapter.py#L607-L609

Problem:
Fast SDXL adapter tests exist, and there are SDXL adapter slow paths in single-file and LoRA integration tests, but `tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py` has no plain slow golden inference test for the default SDXL adapter pipeline.

Impact:
Core SDXL adapter behavior can regress without a direct slow pipeline fixture. The `output_type="latent"` return bug and constructor/list handling are not covered by existing slow SDXL adapter tests.

Reproduction:
```python
from pathlib import Path

sdxl_test = Path("tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py").read_text()
sd_test = Path("tests/pipelines/stable_diffusion_adapter/test_stable_diffusion_adapter.py").read_text()

print("@slow in SDXL adapter pipeline test:", "@slow" in sdxl_test)
print("@slow in SD adapter pipeline test:", "@slow" in sd_test)
```

Relevant precedent:
The SD adapter pipeline has a dedicated slow class with real adapter checkpoints and expected arrays.

Suggested fix:
Add a `@slow` SDXL adapter pipeline regression test in `tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py`, using an `hf-internal-testing` image and a stable expected array under `datasets/diffusers/test-arrays`, covering at least normal inference and `output_type="latent", return_dict=False`.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia da adapter.py e dai file interessati della pipeline dell’adapter Stable Diffusion, quindi esegui le sei riproduzioni indicate nell’issue per confermare ogni errore segnalato. Esamina i precedenti della pipeline SD/SDXL indicati e gli esempi in README_sdxl.md e training/t2i_adapters.md. Il lavoro è completato quando i valori predefiniti dell’adapter elencati, il costruttore, l’output latente, la serializzazione di PathLike, i loader mixin e gli esempi della documentazione si comportano come documentato.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
documentation, machine-learning
Tipo di issue
Bug
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.