huggingface / huggingface/diffusers
t2i_adapter model/pipeline review
- Ngôn ngữ chính
- Python
- Star
- 34.5k
- Fork
- 7.3k
- Merge trung bình
- 3 ngày 3 giờ
- Pull request đã merge (30 ngày)
- 91
Mô tả
# `t2i_adapter` model/pipeline review
Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`
Review performed against the repository review rules.
Reviewed: target model/pipeline files, public exports/lazy imports, serialization/loading, dtype/device/offload paths, related SD/SDXL precedents, fast/slow tests, docs, and examples. Public imports and lazy-loading registration look correct.
Duplicate searches run with `gh search issues/prs` for `t2i_adapter`, affected class names, `MultiAdapter`, `adapter_conditioning_scale`, `iteration over a 0-d tensor`, SDXL list adapters, latent output, PathLike save/load, docs scheduler typo, and slow coverage.
## Issue 1: `MultiAdapter` still breaks on the pipeline default scale
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L88-L94
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L884-L885
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L1166-L1167
Problem:
Both pipelines pass the default `adapter_conditioning_scale=1.0` to `MultiAdapter.forward`. `MultiAdapter.forward` converts that float to a scalar tensor and then iterates it, raising `TypeError: iteration over a 0-d tensor`. It also silently truncates when a scale list has the wrong length.
Duplicate check:
This exact default-scale failure was reported in closed issue https://github.com/huggingface/diffusers/issues/6274 and still reproduces on this commit, so this is not a new finding.
Impact:
A documented/default multi-adapter call fails unless users know to pass a list. Wrong-length scale lists can silently skip adapters.
Reproduction:
```python
import torch
from diffusers import MultiAdapter, T2IAdapter
multi = MultiAdapter([
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
])
xs = [torch.randn(1, 3, 8, 8), torch.randn(1, 3, 8, 8)]
try:
multi(xs, 1.0)
except Exception as e:
print(type(e).__name__, str(e))
print("short list accepted:", multi(xs, [1.0])[0].shape)
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/community/pipeline_stable_diffusion_xl_controlnet_adapter.py#L1089-L1090
Suggested fix:
```python
if adapter_weights is None:
adapter_weights = [1 / self.num_adapter] * self.num_adapter
elif isinstance(adapter_weights, (float, int)):
adapter_weights = [float(adapter_weights)] * self.num_adapter
elif len(adapter_weights) != self.num_adapter:
raise ValueError(
f"`adapter_weights` must have length {self.num_adapter}, got {len(adapter_weights)}."
)
if len(xs) != self.num_adapter:
raise ValueError(f"`xs` must have length {self.num_adapter}, got {len(xs)}.")
```
## Issue 2: SDXL adapter pipeline does not accept `list[T2IAdapter]` despite its public signature
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L273-L290
Problem:
`StableDiffusionXLAdapterPipeline.__init__` documents and types `adapter` as `T2IAdapter | MultiAdapter | list[T2IAdapter]`, but registers the raw list. `register_modules` then fails because a Python list has no `__module__`.
Impact:
SDXL is inconsistent with the SD adapter pipeline and breaks a documented constructor form.
Reproduction:
```python
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter
try:
StableDiffusionXLAdapterPipeline(
vae=None, text_encoder=None, text_encoder_2=None,
tokenizer=None, tokenizer_2=None, unet=None, scheduler=None,
adapter=[
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
],
)
except Exception as e:
print(type(e).__name__, str(e))
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L260-L261
Suggested fix:
```python
if isinstance(adapter, (list, tuple)):
adapter = MultiAdapter(adapter)
self.register_modules(
vae=vae,
text_encoder=text_encoder,
text_encoder_2=text_encoder_2,
tokenizer=tokenizer,
tokenizer_2=tokenizer_2,
unet=unet,
adapter=adapter,
scheduler=scheduler,
feature_extractor=feature_extractor,
image_encoder=image_encoder,
)
```
## Issue 3: SDXL latent output returns before cleanup and ignores `return_dict=False`
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L1278-L1298
Problem:
For `output_type="latent"`, `StableDiffusionXLAdapterPipeline.__call__` returns immediately, before `maybe_free_model_hooks()` and before the `return_dict` handling.
Impact:
Model offload hooks are not released on latent output, and `return_dict=False` still returns `StableDiffusionXLPipelineOutput`.
Reproduction:
```python
import types
import torch
from diffusers import AutoencoderKL, EulerDiscreteScheduler, StableDiffusionXLAdapterPipeline, T2IAdapter, UNet2DConditionModel
unet = UNet2DConditionModel(
block_out_channels=(32, 64), layers_per_block=1, sample_size=32,
in_channels=4, out_channels=4,
down_block_types=("DownBlock2D", "CrossAttnDownBlock2D"),
up_block_types=("CrossAttnUpBlock2D", "UpBlock2D"),
attention_head_dim=(2, 4), use_linear_projection=True,
addition_embed_type="text_time", addition_time_embed_dim=8,
transformer_layers_per_block=(1, 1),
projection_class_embeddings_input_dim=80, cross_attention_dim=64,
)
vae = AutoencoderKL(
block_out_channels=[32, 64], in_channels=3, out_channels=3,
down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"],
up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D"], latent_channels=4,
)
pipe = StableDiffusionXLAdapterPipeline(
vae=vae, text_encoder=None, text_encoder_2=None, tokenizer=None, tokenizer_2=None,
unet=unet,
adapter=T2IAdapter(in_channels=3, channels=[32, 64], num_res_blocks=1, downscale_factor=4, adapter_type="full_adapter_xl"),
scheduler=EulerDiscreteScheduler(),
)
pipe.set_progress_bar_config(disable=True)
pipe.freed = False
pipe.maybe_free_model_hooks = types.MethodType(lambda self: setattr(self, "freed", True), pipe)
out = pipe(
prompt_embeds=torch.zeros(1, 2, 64),
negative_prompt_embeds=torch.zeros(1, 2, 64),
pooled_prompt_embeds=torch.zeros(1, 32),
negative_pooled_prompt_embeds=torch.zeros(1, 32),
image=torch.zeros(1, 3, 64, 64),
num_inference_steps=1,
guidance_scale=1.0,
output_type="latent",
return_dict=False,
)
print(type(out).__name__, pipe.freed)
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py#L1287-L1300
Suggested fix:
```python
else:
image = latents
if not output_type == "latent":
image = self.image_processor.postprocess(image, output_type=output_type)
self.maybe_free_model_hooks()
if not return_dict:
return (image,)
return StableDiffusionXLPipelineOutput(images=image)
```
## Issue 4: `MultiAdapter.save_pretrained` and `from_pretrained` reject `PathLike`
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L130-L145
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/adapter.py#L196-L204
Problem:
The signatures accept `str | os.PathLike`, but the implementation concatenates paths with `+ f"_{idx}"`, which fails for `pathlib.Path`.
Impact:
Serialization/loading works with strings but fails with standard path objects.
Reproduction:
```python
from pathlib import Path
import tempfile
from diffusers import MultiAdapter, T2IAdapter
multi = MultiAdapter([
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
])
with tempfile.TemporaryDirectory() as d:
try:
multi.save_pretrained(Path(d) / "adapter")
except Exception as e:
print("save:", type(e).__name__, str(e))
with tempfile.TemporaryDirectory() as d:
path = Path(d) / "adapter"
multi.save_pretrained(str(path))
try:
MultiAdapter.from_pretrained(path)
except Exception as e:
print("load:", type(e).__name__, str(e))
```
Relevant precedent:
`T2IAdapter` inherits the normal `ModelMixin` path handling; this custom override should preserve the same public contract.
Suggested fix:
```python
save_directory = os.fspath(save_directory)
...
model_path_to_save = f"{save_directory}_{idx}"
pretrained_model_path = os.fspath(pretrained_model_path)
...
model_path_to_load = f"{pretrained_model_path}_{idx}"
```
## Issue 5: SD adapter has dead LoRA/textual-inversion hooks because it does not inherit the loader mixins
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L25
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L191
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L354-L372
Problem:
`StableDiffusionAdapterPipeline` imports `StableDiffusionLoraLoaderMixin` and `TextualInversionLoaderMixin`, and `encode_prompt` checks for them, but the class does not inherit either mixin.
Impact:
`StableDiffusionAdapterPipeline` cannot load LoRA or textual inversion, unlike `StableDiffusionPipeline` and `StableDiffusionXLAdapterPipeline`.
Reproduction:
```python
from diffusers import StableDiffusionAdapterPipeline, StableDiffusionPipeline, StableDiffusionXLAdapterPipeline
for cls in [StableDiffusionPipeline, StableDiffusionAdapterPipeline, StableDiffusionXLAdapterPipeline]:
print(cls.__name__, hasattr(cls, "load_lora_weights"), hasattr(cls, "load_textual_inversion"))
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L154-L160
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L213-L220
Suggested fix:
```python
class StableDiffusionAdapterPipeline(
DiffusionPipeline,
StableDiffusionMixin,
TextualInversionLoaderMixin,
StableDiffusionLoraLoaderMixin,
FromSingleFileMixin,
):
...
```
## Issue 6: T2I-Adapter docs import a nonexistent scheduler class
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/t2i_adapter/README_sdxl.md#L97-L110
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/docs/source/en/training/t2i_adapters.md#L191-L200
Problem:
The inference snippets import `EulerAncestralDiscreteSchedulerTest`, which is not exported. The training docs also assign from `pipe.scheduler.config` while the variable is named `pipeline`.
Impact:
Users following the example hit an immediate import/name error.
Reproduction:
```python
try:
from diffusers import EulerAncestralDiscreteSchedulerTest
except Exception as e:
print(type(e).__name__, str(e))
```
Relevant precedent:
Use the public scheduler class exported by diffusers.
Suggested fix:
```python
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, EulerAncestralDiscreteScheduler
...
pipeline.scheduler = EulerAncestralDiscreteScheduler.from_config(pipeline.scheduler.config)
```
## Issue 7: SDXL adapter lacks a plain slow golden test in its pipeline test file
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py#L52
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/stable_diffusion_adapter/test_stable_diffusion_adapter.py#L607-L609
Problem:
Fast SDXL adapter tests exist, and there are SDXL adapter slow paths in single-file and LoRA integration tests, but `tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py` has no plain slow golden inference test for the default SDXL adapter pipeline.
Impact:
Core SDXL adapter behavior can regress without a direct slow pipeline fixture. The `output_type="latent"` return bug and constructor/list handling are not covered by existing slow SDXL adapter tests.
Reproduction:
```python
from pathlib import Path
sdxl_test = Path("tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py").read_text()
sd_test = Path("tests/pipelines/stable_diffusion_adapter/test_stable_diffusion_adapter.py").read_text()
print("@slow in SDXL adapter pipeline test:", "@slow" in sdxl_test)
print("@slow in SD adapter pipeline test:", "@slow" in sd_test)
```
Relevant precedent:
The SD adapter pipeline has a dedicated slow class with real adapter checkpoints and expected arrays.
Suggested fix:
Add a `@slow` SDXL adapter pipeline regression test in `tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py`, using an `hf-internal-testing` image and a stable expected array under `datasets/diffusers/test-arrays`, covering at least normal inference and `output_type="latent", return_dict=False`.
Hướng dẫn đóng góp
Hướng nghiên cứu
Bắt đầu với adapter.py bị ảnh hưởng và các tệp pipeline của adapter Stable Diffusion, sau đó chạy sáu trường hợp tái hiện trong issue để xác nhận từng lỗi được báo cáo. Xem lại các tiền lệ pipeline SD/SDXL được tham chiếu và các ví dụ trong README_sdxl.md và training/t2i_adapters.md. Được xem là hoàn tất khi các giá trị mặc định của adapter được liệt kê, constructor, đầu ra latent, serialization của PathLike, các loader mixin và các ví dụ trong tài liệu hoạt động như được mô tả.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- python, pytorch
- Lĩnh vực
- documentation, machine-learning
- Loại issue
- Lỗi
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức độ hoạt động
- Ít trao đổi
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 35/100