huggingface / huggingface/diffusers
t2i_adapter model/pipeline review
- Langage dominant
- Python
- Étoiles
- 34.5k
- Forks
- 7.3k
- Merge moyen
- 3 j 3 h
- PR mergées (30 j)
- 91
Description
# `t2i_adapter` model/pipeline review
Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`
Review performed against the repository review rules.
Reviewed: target model/pipeline files, public exports/lazy imports, serialization/loading, dtype/device/offload paths, related SD/SDXL precedents, fast/slow tests, docs, and examples. Public imports and lazy-loading registration look correct.
Duplicate searches run with `gh search issues/prs` for `t2i_adapter`, affected class names, `MultiAdapter`, `adapter_conditioning_scale`, `iteration over a 0-d tensor`, SDXL list adapters, latent output, PathLike save/load, docs scheduler typo, and slow coverage.
## Issue 1: `MultiAdapter` still breaks on the pipeline default scale
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L88-L94
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L884-L885
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L1166-L1167
Problem:
Both pipelines pass the default `adapter_conditioning_scale=1.0` to `MultiAdapter.forward`. `MultiAdapter.forward` converts that float to a scalar tensor and then iterates it, raising `TypeError: iteration over a 0-d tensor`. It also silently truncates when a scale list has the wrong length.
Duplicate check:
This exact default-scale failure was reported in closed issue https://github.com/huggingface/diffusers/issues/6274 and still reproduces on this commit, so this is not a new finding.
Impact:
A documented/default multi-adapter call fails unless users know to pass a list. Wrong-length scale lists can silently skip adapters.
Reproduction:
```python
import torch
from diffusers import MultiAdapter, T2IAdapter
multi = MultiAdapter([
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
])
xs = [torch.randn(1, 3, 8, 8), torch.randn(1, 3, 8, 8)]
try:
multi(xs, 1.0)
except Exception as e:
print(type(e).__name__, str(e))
print("short list accepted:", multi(xs, [1.0])[0].shape)
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/community/pipeline_stable_diffusion_xl_controlnet_adapter.py#L1089-L1090
Suggested fix:
```python
if adapter_weights is None:
adapter_weights = [1 / self.num_adapter] * self.num_adapter
elif isinstance(adapter_weights, (float, int)):
adapter_weights = [float(adapter_weights)] * self.num_adapter
elif len(adapter_weights) != self.num_adapter:
raise ValueError(
f"`adapter_weights` must have length {self.num_adapter}, got {len(adapter_weights)}."
)
if len(xs) != self.num_adapter:
raise ValueError(f"`xs` must have length {self.num_adapter}, got {len(xs)}.")
```
## Issue 2: SDXL adapter pipeline does not accept `list[T2IAdapter]` despite its public signature
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L273-L290
Problem:
`StableDiffusionXLAdapterPipeline.__init__` documents and types `adapter` as `T2IAdapter | MultiAdapter | list[T2IAdapter]`, but registers the raw list. `register_modules` then fails because a Python list has no `__module__`.
Impact:
SDXL is inconsistent with the SD adapter pipeline and breaks a documented constructor form.
Reproduction:
```python
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter
try:
StableDiffusionXLAdapterPipeline(
vae=None, text_encoder=None, text_encoder_2=None,
tokenizer=None, tokenizer_2=None, unet=None, scheduler=None,
adapter=[
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
],
)
except Exception as e:
print(type(e).__name__, str(e))
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L260-L261
Suggested fix:
```python
if isinstance(adapter, (list, tuple)):
adapter = MultiAdapter(adapter)
self.register_modules(
vae=vae,
text_encoder=text_encoder,
text_encoder_2=text_encoder_2,
tokenizer=tokenizer,
tokenizer_2=tokenizer_2,
unet=unet,
adapter=adapter,
scheduler=scheduler,
feature_extractor=feature_extractor,
image_encoder=image_encoder,
)
```
## Issue 3: SDXL latent output returns before cleanup and ignores `return_dict=False`
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L1278-L1298
Problem:
For `output_type="latent"`, `StableDiffusionXLAdapterPipeline.__call__` returns immediately, before `maybe_free_model_hooks()` and before the `return_dict` handling.
Impact:
Model offload hooks are not released on latent output, and `return_dict=False` still returns `StableDiffusionXLPipelineOutput`.
Reproduction:
```python
import types
import torch
from diffusers import AutoencoderKL, EulerDiscreteScheduler, StableDiffusionXLAdapterPipeline, T2IAdapter, UNet2DConditionModel
unet = UNet2DConditionModel(
block_out_channels=(32, 64), layers_per_block=1, sample_size=32,
in_channels=4, out_channels=4,
down_block_types=("DownBlock2D", "CrossAttnDownBlock2D"),
up_block_types=("CrossAttnUpBlock2D", "UpBlock2D"),
attention_head_dim=(2, 4), use_linear_projection=True,
addition_embed_type="text_time", addition_time_embed_dim=8,
transformer_layers_per_block=(1, 1),
projection_class_embeddings_input_dim=80, cross_attention_dim=64,
)
vae = AutoencoderKL(
block_out_channels=[32, 64], in_channels=3, out_channels=3,
down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"],
up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D"], latent_channels=4,
)
pipe = StableDiffusionXLAdapterPipeline(
vae=vae, text_encoder=None, text_encoder_2=None, tokenizer=None, tokenizer_2=None,
unet=unet,
adapter=T2IAdapter(in_channels=3, channels=[32, 64], num_res_blocks=1, downscale_factor=4, adapter_type="full_adapter_xl"),
scheduler=EulerDiscreteScheduler(),
)
pipe.set_progress_bar_config(disable=True)
pipe.freed = False
pipe.maybe_free_model_hooks = types.MethodType(lambda self: setattr(self, "freed", True), pipe)
out = pipe(
prompt_embeds=torch.zeros(1, 2, 64),
negative_prompt_embeds=torch.zeros(1, 2, 64),
pooled_prompt_embeds=torch.zeros(1, 32),
negative_pooled_prompt_embeds=torch.zeros(1, 32),
image=torch.zeros(1, 3, 64, 64),
num_inference_steps=1,
guidance_scale=1.0,
output_type="latent",
return_dict=False,
)
print(type(out).__name__, pipe.freed)
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py#L1287-L1300
Suggested fix:
```python
else:
image = latents
if not output_type == "latent":
image = self.image_processor.postprocess(image, output_type=output_type)
self.maybe_free_model_hooks()
if not return_dict:
return (image,)
return StableDiffusionXLPipelineOutput(images=image)
```
## Issue 4: `MultiAdapter.save_pretrained` and `from_pretrained` reject `PathLike`
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L130-L145
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L196-L204
Problem:
The signatures accept `str | os.PathLike`, but the implementation concatenates paths with `+ f"_{idx}"`, which fails for `pathlib.Path`.
Impact:
Serialization/loading works with strings but fails with standard path objects.
Reproduction:
```python
from pathlib import Path
import tempfile
from diffusers import MultiAdapter, T2IAdapter
multi = MultiAdapter([
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
])
with tempfile.TemporaryDirectory() as d:
try:
multi.save_pretrained(Path(d) / "adapter")
except Exception as e:
print("save:", type(e).__name__, str(e))
with tempfile.TemporaryDirectory() as d:
path = Path(d) / "adapter"
multi.save_pretrained(str(path))
try:
MultiAdapter.from_pretrained(path)
except Exception as e:
print("load:", type(e).__name__, str(e))
```
Relevant precedent:
`T2IAdapter` inherits the normal `ModelMixin` path handling; this custom override should preserve the same public contract.
Suggested fix:
```python
save_directory = os.fspath(save_directory)
...
model_path_to_save = f"{save_directory}_{idx}"
pretrained_model_path = os.fspath(pretrained_model_path)
...
model_path_to_load = f"{pretrained_model_path}_{idx}"
```
## Issue 5: SD adapter has dead LoRA/textual-inversion hooks because it does not inherit the loader mixins
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L25
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L191
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L354-L372
Problem:
`StableDiffusionAdapterPipeline` imports `StableDiffusionLoraLoaderMixin` and `TextualInversionLoaderMixin`, and `encode_prompt` checks for them, but the class does not inherit either mixin.
Impact:
`StableDiffusionAdapterPipeline` cannot load LoRA or textual inversion, unlike `StableDiffusionPipeline` and `StableDiffusionXLAdapterPipeline`.
Reproduction:
```python
from diffusers import StableDiffusionAdapterPipeline, StableDiffusionPipeline, StableDiffusionXLAdapterPipeline
for cls in [StableDiffusionPipeline, StableDiffusionAdapterPipeline, StableDiffusionXLAdapterPipeline]:
print(cls.__name__, hasattr(cls, "load_lora_weights"), hasattr(cls, "load_textual_inversion"))
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L154-L160
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L213-L220
Suggested fix:
```python
class StableDiffusionAdapterPipeline(
DiffusionPipeline,
StableDiffusionMixin,
TextualInversionLoaderMixin,
StableDiffusionLoraLoaderMixin,
FromSingleFileMixin,
):
...
```
## Issue 6: T2I-Adapter docs import a nonexistent scheduler class
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/t2i_adapter/README_sdxl.md#L97-L110
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/docs/source/en/training/t2i_adapters.md#L191-L200
Problem:
The inference snippets import `EulerAncestralDiscreteSchedulerTest`, which is not exported. The training docs also assign from `pipe.scheduler.config` while the variable is named `pipeline`.
Impact:
Users following the example hit an immediate import/name error.
Reproduction:
```python
try:
from diffusers import EulerAncestralDiscreteSchedulerTest
except Exception as e:
print(type(e).__name__, str(e))
```
Relevant precedent:
Use the public scheduler class exported by diffusers.
Suggested fix:
```python
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, EulerAncestralDiscreteScheduler
...
pipeline.scheduler = EulerAncestralDiscreteScheduler.from_config(pipeline.scheduler.config)
```
## Issue 7: SDXL adapter lacks a plain slow golden test in its pipeline test file
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py#L52
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/stable_diffusion_adapter/test_stable_diffusion_adapter.py#L607-L609
Problem:
Fast SDXL adapter tests exist, and there are SDXL adapter slow paths in single-file and LoRA integration tests, but `tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py` has no plain slow golden inference test for the default SDXL adapter pipeline.
Impact:
Core SDXL adapter behavior can regress without a direct slow pipeline fixture. The `output_type="latent"` return bug and constructor/list handling are not covered by existing slow SDXL adapter tests.
Reproduction:
```python
from pathlib import Path
sdxl_test = Path("tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py").read_text()
sd_test = Path("tests/pipelines/stable_diffusion_adapter/test_stable_diffusion_adapter.py").read_text()
print("@slow in SDXL adapter pipeline test:", "@slow" in sdxl_test)
print("@slow in SD adapter pipeline test:", "@slow" in sd_test)
```
Relevant precedent:
The SD adapter pipeline has a dedicated slow class with real adapter checkpoints and expected arrays.
Suggested fix:
Add a `@slow` SDXL adapter pipeline regression test in `tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py`, using an `hf-internal-testing` image and a stable expected array under `datasets/diffusers/test-arrays`, covering at least normal inference and `output_type="latent", return_dict=False`.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par le fichier adapter.py concerné et les fichiers du pipeline de l’adapter Stable Diffusion, puis exécutez les six reproductions de l’issue afin de confirmer chaque échec signalé. Examinez les précédents de pipeline SD/SDXL référencés ainsi que les exemples de README_sdxl.md et training/t2i_adapters.md. Le travail est terminé lorsque les valeurs par défaut de l’adapter listées, le constructeur, la sortie latente, la sérialisation de PathLike, les loader mixins et les exemples de documentation se comportent comme documenté.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- python, pytorch
- Domaine
- documentation, machine-learning
- Type d'issue
- Bug
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Activité
- Calme
- Clarté
- Plutôt claire
- Accessibilité débutants
- 35/100