huggingface / huggingface/diffusers
latent_consistency_models model/pipeline review
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
# `latent_consistency_models` model/pipeline review
Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`
Review performed against the repository review rules.
## Issue 1: Img2Img does not serialize `requires_safety_checker`
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py#L216-L239
Problem:
`LatentConsistencyModelImg2ImgPipeline.__init__` accepts `requires_safety_checker`, but never calls `self.register_to_config(...)`. The text2img LCM pipeline does register it, so img2img saved configs lose this user choice.
Impact:
Pipelines constructed with `requires_safety_checker=False` do not persist that setting through config/save/load paths, causing inconsistent serialization behavior between the two LCM pipelines.
Reproduction:
```python
from diffusers import LatentConsistencyModelPipeline, LatentConsistencyModelImg2ImgPipeline
kwargs = dict(
vae=None,
text_encoder=None,
tokenizer=None,
unet=None,
scheduler=None,
safety_checker=None,
feature_extractor=None,
requires_safety_checker=False,
)
for cls in (LatentConsistencyModelPipeline, LatentConsistencyModelImg2ImgPipeline):
pipe = cls(**kwargs)
print(cls.__name__, pipe.config.get("requires_safety_checker"))
# Text2Img prints False; Img2Img prints None.
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py#L211-L223
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py#L314-L322
Suggested fix:
```python
self.vae_scale_factor = 2 ** (len(self.vae.config.block_out_channels) - 1) if getattr(self, "vae", None) else 8
self.image_processor = VaeImageProcessor(vae_scale_factor=self.vae_scale_factor)
self.register_to_config(requires_safety_checker=requires_safety_checker)
```
Duplicate search:
No matching issue or PR found for `LatentConsistencyModelImg2ImgPipeline requires_safety_checker`.
## Issue 2: Img2Img accepts a safety checker without a feature extractor
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py#L216-L239
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py#L494-L506
Problem:
The img2img constructor does not reject `safety_checker != None` with `feature_extractor=None`. Later, `run_safety_checker` unconditionally calls `self.feature_extractor(...)`, producing a late `TypeError`.
Impact:
Users can construct an invalid pipeline successfully and only fail during inference/safety checking with an unclear `'NoneType' object is not callable` error.
Reproduction:
```python
import torch
from diffusers import LatentConsistencyModelImg2ImgPipeline
class DummySafetyChecker:
def __call__(self, images, clip_input):
return images, [False] * images.shape[0]
pipe = LatentConsistencyModelImg2ImgPipeline(
vae=None,
text_encoder=None,
tokenizer=None,
unet=None,
scheduler=None,
safety_checker=DummySafetyChecker(),
feature_extractor=None,
requires_safety_checker=True,
)
try:
pipe.run_safety_checker(torch.zeros(1, 3, 8, 8), "cpu", torch.float32)
except Exception as e:
print(type(e).__name__, str(e))
# TypeError 'NoneType' object is not callable
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py#L205-L209
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py#L279-L283
Suggested fix:
```python
if safety_checker is not None and feature_extractor is None:
raise ValueError(
"Make sure to define a feature extractor when loading {self.__class__} if you want to use the safety"
" checker. If you do not want to use the safety checker, you can pass `'safety_checker=None'` instead."
)
```
Duplicate search:
No matching issue or PR found for `LatentConsistencyModelImg2ImgPipeline feature_extractor`.
## Issue 3: LCM pipeline `__call__` docs have stale parameters and defaults
Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py#L642-L690
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py#L711-L760
Problem:
Both `__call__` docstrings say `num_inference_steps` defaults to `50` and `guidance_scale` defaults to `7.5`, while the signatures default to `4` and `8.5`. The img2img docstring also documents nonexistent `height` and `width` parameters and omits its real `image` and `strength` parameters.
Impact:
The generated API docs mislead users about LCM's fast-step defaults and img2img inputs.
Reproduction:
```python
import inspect
from diffusers import LatentConsistencyModelPipeline, LatentConsistencyModelImg2ImgPipeline
for cls in (LatentConsistencyModelPipeline, LatentConsistencyModelImg2ImgPipeline):
sig = inspect.signature(cls.__call__)
doc = inspect.getdoc(cls.__call__) or ""
print(cls.__name__, sig.parameters["num_inference_steps"].default, sig.parameters["guidance_scale"].default)
print("doc says steps default 50:", "defaults to 50" in doc)
print("doc says guidance default 7.5:", "defaults to 7.5" in doc)
img_doc = inspect.getdoc(LatentConsistencyModelImg2ImgPipeline.__call__) or ""
print("img2img signature has image:", "image" in inspect.signature(LatentConsistencyModelImg2ImgPipeline.__call__).parameters)
print("img2img docs image:", "image (`" in img_doc)
print("img2img docs height:", "height (`int`" in img_doc)
```
Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py#L891-L902
Suggested fix:
Update the LCM docstrings to match their signatures: `num_inference_steps` default `4`, `guidance_scale` default `8.5`, and for img2img replace the stale `height`/`width` entries with `image` and `strength` documentation.
Duplicate search:
No matching issue or PR found for LCM img2img docstring/default mismatches.
Coverage and duplicate-search status:
Fast and slow tests exist for both LCM pipelines under `tests/pipelines/latent_consistency_models/`; slow coverage is not missing. Local target pytest collection was blocked by the venv torch build missing `torch._C._distributed_c10d` via shared test utilities. `python utils/check_copies.py` passed. Broad duplicate searches found historical LCM issues/PRs, but no duplicates for the three findings above.
Contributor guide
Research direction
Start with src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py and pipeline_latent_consistency_text2img.py, comparing the cited constructors, safety-checker handling, and __call__ docstrings. Run the fast and slow tests under tests/pipelines/latent_consistency_models/ and python utils/check_copies.py; done means config persistence, invalid safety-checker combinations, and both pipelines' documented parameters and defaults are covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- documentation, machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 64/100