latent_diffusion model/pipeline review

# `latent_diffusion` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

Duplicate search: searched GitHub Issues/PRs for `latent_diffusion`, `LDMTextToImagePipeline`, `LDMSuperResolutionPipeline`, `LDMBertConfig`, and the specific failure modes below. I found no matching duplicates; old issues #170/#211 are unrelated historical failures.

## Issue 1: `guidance_scale=0` ignores the prompt

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py#L153-L207

Problem:
CFG is enabled whenever `guidance_scale != 1.0`. Diffusers pipelines document and implement CFG as enabled when `guidance_scale > 1`; at `0` or `0.5`, users expect no CFG, not unconditional-prompt interpolation. With the current branch, `guidance_scale=0` returns the unconditional prediction and removes prompt influence.

Impact:
Low guidance values silently produce prompt-insensitive or under-conditioned results.

Reproduction:
```python
import torch
from types import SimpleNamespace
from diffusers import DDIMScheduler, LDMTextToImagePipeline

class M(torch.nn.Module):
@property
def device(self): return next(self.parameters()).device
@property
def dtype(self): return next(self.parameters()).dtype

class Tok:
def __call__(self, prompt, max_length=None, **_):
prompt = [prompt] if isinstance(prompt, str) else prompt
ids = torch.zeros(len(prompt), max_length, dtype=torch.long)
for i, p in enumerate(prompt):
ids[i, 0] = sum(map(ord, p)) % 100 + 1 if p else 0
return SimpleNamespace(input_ids=ids)

class Text(M):
def __init__(self): super().__init__(); self.p = torch.nn.Parameter(torch.ones(()))
def forward(self, ids): return (ids.float().unsqueeze(-1).repeat(1, 1, 32) / 100,)

class UNet(M):
def __init__(self):
super().__init__(); self.p = torch.nn.Parameter(torch.ones(()))
self.config = SimpleNamespace(in_channels=4, sample_size=8)
def forward(self, sample, timestep, encoder_hidden_states=None):
v = encoder_hidden_states[:, 0, 0].view(-1, 1, 1, 1).to(sample)
return SimpleNamespace(sample=v.expand_as(sample))

class VAE(M):
def __init__(self):
super().__init__(); self.p = torch.nn.Parameter(torch.ones(()))
self.config = SimpleNamespace(block_out_channels=(1,1,1,1), scaling_factor=1.0)
def decode(self, latents):
return SimpleNamespace(sample=latents[:, :3].repeat_interleave(8, -1).repeat_interleave(8, -2))

pipe = LDMTextToImagePipeline(VAE(), Text(), Tok(), UNet(), DDIMScheduler())
pipe.set_progress_bar_config(disable=True)
latents = torch.zeros(1, 4, 8, 8)
run = lambda p, gs: pipe(p, latents=latents.clone(), num_inference_steps=1, guidance_scale=gs, output_type="np").images
print(abs(run("cat", 0.0) - run("dog", 0.0)).max()) # 0.0: prompt ignored
print(abs(run("cat", 1.0) - run("dog", 1.0)).max()) # prompt affects output
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L763-L764

Suggested fix:
```python
do_classifier_free_guidance = guidance_scale > 1.0
if do_classifier_free_guidance:
...
if do_classifier_free_guidance:
latents_input = torch.cat([latents] * 2)
context = torch.cat([negative_prompt_embeds, prompt_embeds])
else:
latents_input = latents
context = prompt_embeds
```

## Issue 2: text-to-image hard-codes latent scale factor `8`

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py#L139-L165

Problem:
`self.vae_scale_factor` is computed in `__init__`, but validation and latent shape still use hard-coded `8`. Tiny VAEs or any compatible `AutoencoderKL`/`VQModel` with a different scale factor generate the wrong output size or reject valid dimensions.

Impact:
The pipeline is inconsistent with its own config-derived scale factor and with fast-test-sized components.

Reproduction:
```python
# Same tiny component pattern as above, but VAE has scale factor 2.
# The pipeline requests default height 16, but creates latents at height//8 and decodes to 4.
print(pipe.vae_scale_factor)
print(pipe.unet.config.sample_size * pipe.vae_scale_factor)
print(pipe("x", num_inference_steps=1, guidance_scale=1.0, output_type="np").images.shape)
# observed shape: (1, 4, 4, 3), expected height/width: 16
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L694-L700

Suggested fix:
```python
if height % self.vae_scale_factor != 0 or width % self.vae_scale_factor != 0:
raise ValueError(
f"`height` and `width` have to be divisible by {self.vae_scale_factor} but are {height} and {width}."
)

latents_shape = (
batch_size,
self.unet.config.in_channels,
height // self.vae_scale_factor,
width // self.vae_scale_factor,
)
```

## Issue 3: LMS scheduler support is incomplete in text-to-image

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py#L172-L210

Problem:
The docstring advertises `LMSDiscreteScheduler`, but `__call__` never scales initial noise by `scheduler.init_noise_sigma` and never calls `scheduler.scale_model_input(...)`. `LMSDiscreteScheduler.step()` emits the standard warning that `scale_model_input` was skipped.

Impact:
Schedulers whose model input scaling is non-noop are driven outside their expected contract, causing incorrect denoising.

Reproduction:
```python
import warnings
# Build the same tiny pipeline as Issue 1, but use LMSDiscreteScheduler.
from diffusers import LMSDiscreteScheduler
pipe.scheduler = LMSDiscreteScheduler()
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
pipe("x", num_inference_steps=4, guidance_scale=1.0, output_type="np")
print([str(w.message) for w in caught if "scale_model_input" in str(w.message)])
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py#L161-L175
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L713-L713
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L1037-L1039

Suggested fix:
```python
self.scheduler.set_timesteps(num_inference_steps, device=self._execution_device)
...
latents = latents * self.scheduler.init_noise_sigma
...
latents_input = self.scheduler.scale_model_input(latents_input, t)
noise_pred = self.unet(latents_input, t, encoder_hidden_states=context).sample
```

## Issue 4: super-resolution cannot use model CPU offload

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py#L39-L69
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py#L152-L157

Problem:
`LDMSuperResolutionPipeline` does not set `model_cpu_offload_seq`, so `enable_model_cpu_offload()` always raises. The runtime path also uses `self.device` instead of `_execution_device`, which would create tensors on CPU even after offload hooks are added.

Impact:
A large published super-resolution pipeline cannot use the normal low-memory offload path.

Reproduction:
```python
from diffusers import DDIMScheduler, LDMSuperResolutionPipeline, UNet2DModel, VQModel

unet = UNet2DModel(sample_size=32, in_channels=6, out_channels=3, block_out_channels=(32, 64))
vqvae = VQModel(in_channels=3, out_channels=3, latent_channels=3, block_out_channels=(32, 64))
pipe = LDMSuperResolutionPipeline(vqvae=vqvae, unet=unet, scheduler=DDIMScheduler())

try:
pipe.enable_model_cpu_offload(device="cpu")
except Exception as e:
print(type(e).__name__, e)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_upscale.py#L114-L114

Suggested fix:
```python
class LDMSuperResolutionPipeline(DiffusionPipeline):
model_cpu_offload_seq = "unet->vqvae"

...
device = self._execution_device
latents = randn_tensor(latents_shape, generator=generator, device=device, dtype=latents_dtype)
image = image.to(device=device, dtype=latents_dtype)
self.scheduler.set_timesteps(num_inference_steps, device=device)
```

## Issue 5: `LDMBertConfig` is defined but not exported by lazy imports

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_diffusion/__init__.py#L24-L37
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py#L252-L296

Problem:
`LDMBertModel` is exported from `diffusers.pipelines.latent_diffusion`, but its matching `LDMBertConfig` is not. Users and conversion utilities must import from the private file path instead of the package.

Impact:
Public import behavior is inconsistent for the model/config pair.

Reproduction:
```python
from diffusers.pipelines.latent_diffusion import LDMBertModel
print(LDMBertModel.__name__)

from diffusers.pipelines.latent_diffusion import LDMBertConfig
# ImportError: cannot import name 'LDMBertConfig'
```

Relevant precedent:
Model/config pairs elsewhere in diffusers are exported together from their owning package.

Suggested fix:
```python
_import_structure["pipeline_latent_diffusion"] = ["LDMBertConfig", "LDMBertModel", "LDMTextToImagePipeline"]

...
from .pipeline_latent_diffusion import LDMBertConfig, LDMBertModel, LDMTextToImagePipeline
```

## Issue 6: no `@slow` coverage for the family

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/latent_diffusion/test_latent_diffusion.py#L139-L182
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/latent_diffusion/test_latent_diffusion_superresolution.py#L119-L133

Problem:
The target has fast tests and nightly tests, but no `@slow` tests. `LDMTextToImagePipelineSlowTests` is named “Slow” but decorated with `@nightly`, and super-resolution only has `@nightly` integration coverage.

Impact:
`RUN_SLOW` does not exercise either published checkpoint path, despite the family having slow/nightly-only behavior not covered by fast tests.

Reproduction:
```python
from pathlib import Path

for path in [
"tests/pipelines/latent_diffusion/test_latent_diffusion.py",
"tests/pipelines/latent_diffusion/test_latent_diffusion_superresolution.py",
]:
text = Path(path).read_text()
print(path, "@slow" in text, "@nightly" in text)
```

Relevant precedent:
Other pipeline integration suites use `@slow` for normal checkpoint regression tests and reserve `@nightly` for heavier/full-output cases.

Suggested fix:
```python
from ...testing_utils import slow

@slow
@require_torch_accelerator
class LDMTextToImagePipelineSlowTests(unittest.TestCase):
...

@slow
@require_torch
class LDMSuperResolutionPipelineSlowTests(unittest.TestCase):
...
```

Verification: `LDMSuperResolutionPipelineFastTests::test_inference_superresolution` passed locally. The text-to-image fast test could not be collected in this `.venv` because the local Torch build is missing `torch._C._distributed_c10d`, imported through the shared pipeline test mixin.

Start with src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py, pipeline_latent_diffusion_superresolution.py, and their __init__.py exports, comparing the cited stable-diffusion and super-resolution implementations. Run the reproductions and the fast tests in tests/pipelines/latent_diffusion/test_latent_diffusion.py and test_latent_diffusion_superresolution.py. Done means the six reported behaviors are corrected and the appropriate slow coverage exists without breaking the existing tests.

Written by the indexing model from the issue text.