huggingface / huggingface/diffusers
t2i_adapter model/pipeline review
- Lingua principale
- Python
- Stelle
- 34.5k
- Fork
- 7.3k
- Merge medio
- 3g 3h
- PR unite (30g)
- 91
Descrizione
# `t2i_adapter` model/pipeline review
Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`
Review performed against the repository review rules.
Reviewed: target model/pipeline files, public exports/lazy imports, serialization/loading, dtype/device/offload paths, related SD/SDXL precedents, fast/slow tests, docs, and examples. Public imports and lazy-loading registration look correct.
Duplicate searches run with `gh search issues/prs` for `t2i_adapter`, affected class names, `MultiAdapter`, `adapter_conditioning_scale`, `iteration over a 0-d tensor`, SDXL list adapters, latent output, PathLike save/load, docs scheduler typo, and slow coverage.
## Issue 1: `MultiAdapter` still breaks on the pipeline default scale
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L88-L94
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L884-L885
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L1166-L1167
Problem:
Both pipelines pass the default `adapter_conditioning_scale=1.0` to `MultiAdapter.forward`. `MultiAdapter.forward` converts that float to a scalar tensor and then iterates it, raising `TypeError: iteration over a 0-d tensor`. It also silently truncates when a scale list has the wrong length.
Duplicate check:
This exact default-scale failure was reported in closed issue https://github.com/huggingface/diffusers/issues/6274 and still reproduces on this commit, so this is not a new finding.
Impact:
A documented/default multi-adapter call fails unless users know to pass a list. Wrong-length scale lists can silently skip adapters.
Reproduction:
```python
import torch
from diffusers import MultiAdapter, T2IAdapter
multi = MultiAdapter([
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
])
xs = [torch.randn(1, 3, 8, 8), torch.randn(1, 3, 8, 8)]
try:
multi(xs, 1.0)
except Exception as e:
print(type(e).__name__, str(e))
print("short list accepted:", multi(xs, [1.0])[0].shape)
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/community/pipeline_stable_diffusion_xl_controlnet_adapter.py#L1089-L1090
Suggested fix:
```python
if adapter_weights is None:
adapter_weights = [1 / self.num_adapter] * self.num_adapter
elif isinstance(adapter_weights, (float, int)):
adapter_weights = [float(adapter_weights)] * self.num_adapter
elif len(adapter_weights) != self.num_adapter:
raise ValueError(
f"`adapter_weights` must have length {self.num_adapter}, got {len(adapter_weights)}."
)
if len(xs) != self.num_adapter:
raise ValueError(f"`xs` must have length {self.num_adapter}, got {len(xs)}.")
```
## Issue 2: SDXL adapter pipeline does not accept `list[T2IAdapter]` despite its public signature
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L273-L290
Problem:
`StableDiffusionXLAdapterPipeline.__init__` documents and types `adapter` as `T2IAdapter | MultiAdapter | list[T2IAdapter]`, but registers the raw list. `register_modules` then fails because a Python list has no `__module__`.
Impact:
SDXL is inconsistent with the SD adapter pipeline and breaks a documented constructor form.
Reproduction:
```python
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter
try:
StableDiffusionXLAdapterPipeline(
vae=None, text_encoder=None, text_encoder_2=None,
tokenizer=None, tokenizer_2=None, unet=None, scheduler=None,
adapter=[
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
],
)
except Exception as e:
print(type(e).__name__, str(e))
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L260-L261
Suggested fix:
```python
if isinstance(adapter, (list, tuple)):
adapter = MultiAdapter(adapter)
self.register_modules(
vae=vae,
text_encoder=text_encoder,
text_encoder_2=text_encoder_2,
tokenizer=tokenizer,
tokenizer_2=tokenizer_2,
unet=unet,
adapter=adapter,
scheduler=scheduler,
feature_extractor=feature_extractor,
image_encoder=image_encoder,
)
```
## Issue 3: SDXL latent output returns before cleanup and ignores `return_dict=False`
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L1278-L1298
Problem:
For `output_type="latent"`, `StableDiffusionXLAdapterPipeline.__call__` returns immediately, before `maybe_free_model_hooks()` and before the `return_dict` handling.
Impact:
Model offload hooks are not released on latent output, and `return_dict=False` still returns `StableDiffusionXLPipelineOutput`.
Reproduction:
```python
import types
import torch
from diffusers import AutoencoderKL, EulerDiscreteScheduler, StableDiffusionXLAdapterPipeline, T2IAdapter, UNet2DConditionModel
unet = UNet2DConditionModel(
block_out_channels=(32, 64), layers_per_block=1, sample_size=32,
in_channels=4, out_channels=4,
down_block_types=("DownBlock2D", "CrossAttnDownBlock2D"),
up_block_types=("CrossAttnUpBlock2D", "UpBlock2D"),
attention_head_dim=(2, 4), use_linear_projection=True,
addition_embed_type="text_time", addition_time_embed_dim=8,
transformer_layers_per_block=(1, 1),
projection_class_embeddings_input_dim=80, cross_attention_dim=64,
)
vae = AutoencoderKL(
block_out_channels=[32, 64], in_channels=3, out_channels=3,
down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"],
up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D"], latent_channels=4,
)
pipe = StableDiffusionXLAdapterPipeline(
vae=vae, text_encoder=None, text_encoder_2=None, tokenizer=None, tokenizer_2=None,
unet=unet,
adapter=T2IAdapter(in_channels=3, channels=[32, 64], num_res_blocks=1, downscale_factor=4, adapter_type="full_adapter_xl"),
scheduler=EulerDiscreteScheduler(),
)
pipe.set_progress_bar_config(disable=True)
pipe.freed = False
pipe.maybe_free_model_hooks = types.MethodType(lambda self: setattr(self, "freed", True), pipe)
out = pipe(
prompt_embeds=torch.zeros(1, 2, 64),
negative_prompt_embeds=torch.zeros(1, 2, 64),
pooled_prompt_embeds=torch.zeros(1, 32),
negative_pooled_prompt_embeds=torch.zeros(1, 32),
image=torch.zeros(1, 3, 64, 64),
num_inference_steps=1,
guidance_scale=1.0,
output_type="latent",
return_dict=False,
)
print(type(out).__name__, pipe.freed)
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py#L1287-L1300
Suggested fix:
```python
else:
image = latents
if not output_type == "latent":
image = self.image_processor.postprocess(image, output_type=output_type)
self.maybe_free_model_hooks()
if not return_dict:
return (image,)
return StableDiffusionXLPipelineOutput(images=image)
```
## Issue 4: `MultiAdapter.save_pretrained` and `from_pretrained` reject `PathLike`
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L130-L145
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L196-L204
Problem:
The signatures accept `str | os.PathLike`, but the implementation concatenates paths with `+ f"_{idx}"`, which fails for `pathlib.Path`.
Impact:
Serialization/loading works with strings but fails with standard path objects.
Reproduction:
```python
from pathlib import Path
import tempfile
from diffusers import MultiAdapter, T2IAdapter
multi = MultiAdapter([
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
])
with tempfile.TemporaryDirectory() as d:
try:
multi.save_pretrained(Path(d) / "adapter")
except Exception as e:
print("save:", type(e).__name__, str(e))
with tempfile.TemporaryDirectory() as d:
path = Path(d) / "adapter"
multi.save_pretrained(str(path))
try:
MultiAdapter.from_pretrained(path)
except Exception as e:
print("load:", type(e).__name__, str(e))
```
Relevant precedent:
`T2IAdapter` inherits the normal `ModelMixin` path handling; this custom override should preserve the same public contract.
Suggested fix:
```python
save_directory = os.fspath(save_directory)
...
model_path_to_save = f"{save_directory}_{idx}"
pretrained_model_path = os.fspath(pretrained_model_path)
...
model_path_to_load = f"{pretrained_model_path}_{idx}"
```
## Issue 5: SD adapter has dead LoRA/textual-inversion hooks because it does not inherit the loader mixins
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L25
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L191
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L354-L372
Problem:
`StableDiffusionAdapterPipeline` imports `StableDiffusionLoraLoaderMixin` and `TextualInversionLoaderMixin`, and `encode_prompt` checks for them, but the class does not inherit either mixin.
Impact:
`StableDiffusionAdapterPipeline` cannot load LoRA or textual inversion, unlike `StableDiffusionPipeline` and `StableDiffusionXLAdapterPipeline`.
Reproduction:
```python
from diffusers import StableDiffusionAdapterPipeline, StableDiffusionPipeline, StableDiffusionXLAdapterPipeline
for cls in [StableDiffusionPipeline, StableDiffusionAdapterPipeline, StableDiffusionXLAdapterPipeline]:
print(cls.__name__, hasattr(cls, "load_lora_weights"), hasattr(cls, "load_textual_inversion"))
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L154-L160
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L213-L220
Suggested fix:
```python
class StableDiffusionAdapterPipeline(
DiffusionPipeline,
StableDiffusionMixin,
TextualInversionLoaderMixin,
StableDiffusionLoraLoaderMixin,
FromSingleFileMixin,
):
...
```
## Issue 6: T2I-Adapter docs import a nonexistent scheduler class
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/t2i_adapter/README_sdxl.md#L97-L110
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/docs/source/en/training/t2i_adapters.md#L191-L200
Problem:
The inference snippets import `EulerAncestralDiscreteSchedulerTest`, which is not exported. The training docs also assign from `pipe.scheduler.config` while the variable is named `pipeline`.
Impact:
Users following the example hit an immediate import/name error.
Reproduction:
```python
try:
from diffusers import EulerAncestralDiscreteSchedulerTest
except Exception as e:
print(type(e).__name__, str(e))
```
Relevant precedent:
Use the public scheduler class exported by diffusers.
Suggested fix:
```python
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, EulerAncestralDiscreteScheduler
...
pipeline.scheduler = EulerAncestralDiscreteScheduler.from_config(pipeline.scheduler.config)
```
## Issue 7: SDXL adapter lacks a plain slow golden test in its pipeline test file
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py#L52
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/stable_diffusion_adapter/test_stable_diffusion_adapter.py#L607-L609
Problem:
Fast SDXL adapter tests exist, and there are SDXL adapter slow paths in single-file and LoRA integration tests, but `tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py` has no plain slow golden inference test for the default SDXL adapter pipeline.
Impact:
Core SDXL adapter behavior can regress without a direct slow pipeline fixture. The `output_type="latent"` return bug and constructor/list handling are not covered by existing slow SDXL adapter tests.
Reproduction:
```python
from pathlib import Path
sdxl_test = Path("tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py").read_text()
sd_test = Path("tests/pipelines/stable_diffusion_adapter/test_stable_diffusion_adapter.py").read_text()
print("@slow in SDXL adapter pipeline test:", "@slow" in sdxl_test)
print("@slow in SD adapter pipeline test:", "@slow" in sd_test)
```
Relevant precedent:
The SD adapter pipeline has a dedicated slow class with real adapter checkpoints and expected arrays.
Suggested fix:
Add a `@slow` SDXL adapter pipeline regression test in `tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py`, using an `hf-internal-testing` image and a stable expected array under `datasets/diffusers/test-arrays`, covering at least normal inference and `output_type="latent", return_dict=False`.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia da adapter.py e dai file interessati della pipeline dell’adapter Stable Diffusion, quindi esegui le sei riproduzioni indicate nell’issue per confermare ogni errore segnalato. Esamina i precedenti della pipeline SD/SDXL indicati e gli esempi in README_sdxl.md e training/t2i_adapters.md. Il lavoro è completato quando i valori predefiniti dell’adapter elencati, il costruttore, l’output latente, la serializzazione di PathLike, i loader mixin e gli esempi della documentazione si comportano come documentato.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python, pytorch
- Ambito
- documentation, machine-learning
- Tipo di issue
- Bug
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 35/100