huggingface / huggingface/diffusers

latent_consistency_models model/pipeline review

Đang mở
#13,637 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
34.5k
Fork
7.3k
Merge trung bình
3 ngày 3 giờ
Pull request đã merge (30 ngày)
91

Mô tả

# `latent_consistency_models` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

## Issue 1: Img2Img does not serialize `requires_safety_checker`

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py#L216-L239

Problem:
`LatentConsistencyModelImg2ImgPipeline.__init__` accepts `requires_safety_checker`, but never calls `self.register_to_config(...)`. The text2img LCM pipeline does register it, so img2img saved configs lose this user choice.

Impact:
Pipelines constructed with `requires_safety_checker=False` do not persist that setting through config/save/load paths, causing inconsistent serialization behavior between the two LCM pipelines.

Reproduction:
```python
from diffusers import LatentConsistencyModelPipeline, LatentConsistencyModelImg2ImgPipeline

kwargs = dict(
vae=None,
text_encoder=None,
tokenizer=None,
unet=None,
scheduler=None,
safety_checker=None,
feature_extractor=None,
requires_safety_checker=False,
)

for cls in (LatentConsistencyModelPipeline, LatentConsistencyModelImg2ImgPipeline):
pipe = cls(**kwargs)
print(cls.__name__, pipe.config.get("requires_safety_checker"))
# Text2Img prints False; Img2Img prints None.
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py#L211-L223
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py#L314-L322

Suggested fix:
```python
self.vae_scale_factor = 2 ** (len(self.vae.config.block_out_channels) - 1) if getattr(self, "vae", None) else 8
self.image_processor = VaeImageProcessor(vae_scale_factor=self.vae_scale_factor)
self.register_to_config(requires_safety_checker=requires_safety_checker)
```

Duplicate search:
No matching issue or PR found for `LatentConsistencyModelImg2ImgPipeline requires_safety_checker`.

## Issue 2: Img2Img accepts a safety checker without a feature extractor

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py#L216-L239
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py#L494-L506

Problem:
The img2img constructor does not reject `safety_checker != None` with `feature_extractor=None`. Later, `run_safety_checker` unconditionally calls `self.feature_extractor(...)`, producing a late `TypeError`.

Impact:
Users can construct an invalid pipeline successfully and only fail during inference/safety checking with an unclear `'NoneType' object is not callable` error.

Reproduction:
```python
import torch
from diffusers import LatentConsistencyModelImg2ImgPipeline

class DummySafetyChecker:
def __call__(self, images, clip_input):
return images, [False] * images.shape[0]

pipe = LatentConsistencyModelImg2ImgPipeline(
vae=None,
text_encoder=None,
tokenizer=None,
unet=None,
scheduler=None,
safety_checker=DummySafetyChecker(),
feature_extractor=None,
requires_safety_checker=True,
)

try:
pipe.run_safety_checker(torch.zeros(1, 3, 8, 8), "cpu", torch.float32)
except Exception as e:
print(type(e).__name__, str(e))
# TypeError 'NoneType' object is not callable
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py#L205-L209
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py#L279-L283

Suggested fix:
```python
if safety_checker is not None and feature_extractor is None:
raise ValueError(
"Make sure to define a feature extractor when loading {self.__class__} if you want to use the safety"
" checker. If you do not want to use the safety checker, you can pass `'safety_checker=None'` instead."
)
```

Duplicate search:
No matching issue or PR found for `LatentConsistencyModelImg2ImgPipeline feature_extractor`.

## Issue 3: LCM pipeline `__call__` docs have stale parameters and defaults

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py#L642-L690
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py#L711-L760

Problem:
Both `__call__` docstrings say `num_inference_steps` defaults to `50` and `guidance_scale` defaults to `7.5`, while the signatures default to `4` and `8.5`. The img2img docstring also documents nonexistent `height` and `width` parameters and omits its real `image` and `strength` parameters.

Impact:
The generated API docs mislead users about LCM's fast-step defaults and img2img inputs.

Reproduction:
```python
import inspect
from diffusers import LatentConsistencyModelPipeline, LatentConsistencyModelImg2ImgPipeline

for cls in (LatentConsistencyModelPipeline, LatentConsistencyModelImg2ImgPipeline):
sig = inspect.signature(cls.__call__)
doc = inspect.getdoc(cls.__call__) or ""
print(cls.__name__, sig.parameters["num_inference_steps"].default, sig.parameters["guidance_scale"].default)
print("doc says steps default 50:", "defaults to 50" in doc)
print("doc says guidance default 7.5:", "defaults to 7.5" in doc)

img_doc = inspect.getdoc(LatentConsistencyModelImg2ImgPipeline.__call__) or ""
print("img2img signature has image:", "image" in inspect.signature(LatentConsistencyModelImg2ImgPipeline.__call__).parameters)
print("img2img docs image:", "image (`" in img_doc)
print("img2img docs height:", "height (`int`" in img_doc)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py#L891-L902

Suggested fix:
Update the LCM docstrings to match their signatures: `num_inference_steps` default `4`, `guidance_scale` default `8.5`, and for img2img replace the stale `height`/`width` entries with `image` and `strength` documentation.

Duplicate search:
No matching issue or PR found for LCM img2img docstring/default mismatches.

Coverage and duplicate-search status:
Fast and slow tests exist for both LCM pipelines under `tests/pipelines/latent_consistency_models/`; slow coverage is not missing. Local target pytest collection was blocked by the venv torch build missing `torch._C._distributed_c10d` via shared test utilities. `python utils/check_copies.py` passed. Broad duplicate searches found historical LCM issues/PRs, but no duplicates for the three findings above.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Bắt đầu với src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py và pipeline_latent_consistency_text2img.py, so sánh các constructor được nêu, cách xử lý safety-checker và các docstring của __call__. Chạy các test nhanh và chậm trong tests/pipelines/latent_consistency_models/ cùng với python utils/check_copies.py; được xem là hoàn thành khi đã bao phủ việc duy trì cấu hình, các tổ hợp safety-checker không hợp lệ và các tham số cùng giá trị mặc định được tài liệu hóa của cả hai pipeline.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python, pytorch
Lĩnh vực
documentation, machine-learning, testing-qa
Loại issue
Lỗi
Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Đặc tả rõ ràng
Mức phù hợp với người mới
64/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.