cogview4 model/pipeline review

# `cogview4` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

Duplicate-search status: `gh search` was API-rate-limited, so I used the GitHub connector searches for `cogview4`, affected classes/files, `attention_mask`, `dispatch_attention_fn`, `sigmas`, `callback_on_step_end`, `CogView4PipelineOutput`, `CogView4ControlPipeline load_lora_weights`, and test coverage. I found no exact duplicates for the findings below. Related: https://github.com/huggingface/diffusers/issues/10962 and https://github.com/huggingface/diffusers/pull/10966 fixed the same prompt/negative embed shape problem for the base pipeline, not the control pipeline. The cache-context part of Issue 3 is already broadly tracked by https://github.com/huggingface/diffusers/issues/12760.

## Issue 1: Text attention masks do not actually mask tokens

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_cogview4.py#L168-L179
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4.py#L624-L648
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4_control.py#L674-L694
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/cogview4-control/train_control_cogview4.py#L1048-L1055

Problem:
`CogView4AttnProcessor` converts the text mask to `query.dtype` before passing it to SDPA. Float SDPA masks are additive bias masks, so `0.0` does not block masked positions. The pipelines also never pass text masks to the transformer, even though `_get_glm_embeds` pads prompts to a multiple of 16. The training script separately builds an unpadded tokenizer mask, so its mask length can disagree with padded `prompt_embeds`.

Impact:
Padded or explicitly masked text tokens can affect image tokens. Batched prompts with different lengths can produce different results from single-prompt inference, and training can fail or train with invalid masks.

Reproduction:
```python
import torch
from diffusers import CogView4Transformer2DModel

torch.manual_seed(0)
model = CogView4Transformer2DModel(
patch_size=2, in_channels=4, out_channels=4, num_layers=1,
attention_head_dim=4, num_attention_heads=2, text_embed_dim=8,
time_embed_dim=8, condition_dim=4,
).eval()

hidden_states = torch.randn(1, 4, 8, 8)
encoder_hidden_states = torch.randn(1, 4, 8)
mask = torch.tensor([[0, 1, 1, 1]])

poisoned = encoder_hidden_states.clone()
poisoned[:, 0] = 1_000_000.0
cleaned = encoder_hidden_states.clone()
cleaned[:, 0] = 0.0

kwargs = dict(
hidden_states=hidden_states,
timestep=torch.tensor([1]),
original_size=torch.tensor([[64, 64]]),
target_size=torch.tensor([[64, 64]]),
crop_coords=torch.tensor([[0, 0]]),
attention_mask=mask,
return_dict=False,
)

with torch.no_grad():
out_poisoned = model(encoder_hidden_states=poisoned, **kwargs)[0]
out_cleaned = model(encoder_hidden_states=cleaned, **kwargs)[0]

leak = (out_poisoned - out_cleaned).abs().max().item()
assert leak < 1e-6, f"masked token leaked into image output: {leak}"
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/qwenimage/pipeline_qwenimage.py#L202-L224
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/qwenimage/pipeline_qwenimage.py#L694-L704
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_qwenimage.py#L946-L952

Suggested fix:
```python
# Keep this as bool. Do not cast to query.dtype.
attention_mask = (attn_mask_matrix > 0).unsqueeze(1)
```
Also return padded attention masks from `_get_glm_embeds` / `encode_prompt`, repeat them with `num_images_per_prompt`, and pass the conditional and unconditional masks to `self.transformer(...)`. The training script should pad `attention_mask` with leading zeros using the same `pad_length` logic as `_get_glm_embeds`.

## Issue 2: Attention backend selection is a no-op for CogView4

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_cogview4.py#L114-L124
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_cogview4.py#L178-L179
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_cogview4.py#L401-L402

Problem:
The CogView4 processors call `F.scaled_dot_product_attention` directly and do not define `_attention_backend` / `_parallel_config`. `model.set_attention_backend(...)` therefore cannot steer CogView4 attention to the requested backend, contrary to the review rules.

Impact:
Users cannot opt into supported diffusers attention backends for CogView4, and context-parallel/backend validation cannot reason about this model correctly.

Reproduction:
```python
from diffusers import CogView4Transformer2DModel

model = CogView4Transformer2DModel(
patch_size=2, in_channels=4, out_channels=4, num_layers=1,
attention_head_dim=4, num_attention_heads=2, text_embed_dim=8,
time_embed_dim=8, condition_dim=4,
)
model.set_attention_backend("native")
processor = model.transformer_blocks[0].attn1.processor

assert hasattr(processor, "_attention_backend"), "CogView4 processor ignores set_attention_backend()"
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_flux.py#L75-L123
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_qwenimage.py#L490-L569

Suggested fix:
```python
from ..attention_dispatch import dispatch_attention_fn

class CogView4AttnProcessor:
_attention_backend = None
_parallel_config = None

...
hidden_states = dispatch_attention_fn(
query,
key,
value,
attn_mask=attention_mask,
dropout_p=0.0,
is_causal=False,
backend=self._attention_backend,
parallel_config=self._parallel_config,
)
```
Apply the same pattern to `CogView4TrainingAttnProcessor`.

## Issue 3: Custom `sigmas` are unusable

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4.py#L104-L110
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4.py#L587-L605
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4_control.py#L637-L655

Problem:
When users pass `sigmas`, the pipelines still synthesize a default `timesteps` array of length `num_inference_steps` and pass both arrays to `retrieve_timesteps`. A short custom sigma schedule therefore fails before inference.

Impact:
The public `sigmas` argument is effectively broken unless users also arrange matching timesteps and `num_inference_steps`, which contradicts the docstring.

Reproduction:
```python
import numpy as np
from diffusers import FlowMatchEulerDiscreteScheduler
from diffusers.pipelines.cogview4.pipeline_cogview4 import retrieve_timesteps

scheduler = FlowMatchEulerDiscreteScheduler(use_dynamic_shifting=True)
num_inference_steps = 50
sigmas = [1.0, 0.5]
timesteps = np.linspace(scheduler.config.num_train_timesteps, 1.0, num_inference_steps)
timesteps = timesteps.astype(np.int64).astype(np.float32)

retrieve_timesteps(scheduler, num_inference_steps, "cpu", timesteps, sigmas, mu=0.5)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/qwenimage/pipeline_qwenimage.py#L67-L105

Suggested fix:
```python
if timesteps is None and sigmas is None:
timesteps = np.linspace(self.scheduler.config.num_train_timesteps, 1.0, num_inference_steps)
timesteps = timesteps.astype(np.int64).astype(np.float32)
elif timesteps is not None:
timesteps = np.array(timesteps).astype(np.int64).astype(np.float32)

if sigmas is None and timesteps is not None:
sigmas = timesteps / self.scheduler.config.num_train_timesteps

timesteps, num_inference_steps = retrieve_timesteps(
self.scheduler, num_inference_steps, timestep_device, timesteps, sigmas, mu=mu
)
```

## Issue 4: Step-end callbacks receive sigma instead of timestep

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4.py#L656-L660
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4_control.py#L703-L707

Problem:
Both pipelines call `callback_on_step_end(self, i, self.scheduler.sigmas[i], ...)`. The callback API and other pipelines pass the current timestep `t`.

Impact:
User callbacks that inspect timestep values, implement custom stopping, or coordinate with scheduler timesteps receive the wrong quantity.

Reproduction:
```python
import torch
from diffusers import AutoencoderKL, CogView4Pipeline, CogView4Transformer2DModel, FlowMatchEulerDiscreteScheduler

transformer = CogView4Transformer2DModel(
patch_size=2, in_channels=4, out_channels=4, num_layers=1,
attention_head_dim=4, num_attention_heads=2, text_embed_dim=8,
time_embed_dim=8, condition_dim=4, sample_size=8,
)
vae = AutoencoderKL(
block_out_channels=[32, 64], in_channels=3, out_channels=3,
down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"],
up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D"],
latent_channels=4, sample_size=32,
)
scheduler = FlowMatchEulerDiscreteScheduler(use_dynamic_shifting=True)
pipe = CogView4Pipeline(None, None, vae, transformer, scheduler)
pipe.set_progress_bar_config(disable=True)

seen = []
def callback(pipe, i, t, kwargs):
seen.append((float(t), float(pipe.scheduler.timesteps[i])))
return kwargs

pipe(
prompt_embeds=torch.randn(1, 4, 8),
num_inference_steps=2,
guidance_scale=1.0,
height=16,
width=16,
output_type="latent",
callback_on_step_end=callback,
)
assert seen[0][0] == seen[0][1], seen
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/qwenimage/pipeline_qwenimage.py#L731-L737
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux.py#L983-L989

Suggested fix:
```python
callback_outputs = callback_on_step_end(self, i, t, callback_kwargs)
```

## Issue 5: Control pipeline is missing base CogView4 pipeline capabilities

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4.py#L137
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4_control.py#L139
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4_control.py#L399-L404
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4_control.py#L674-L694
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/cogview4-control/README.md#L99-L123

Problem:
`CogView4ControlPipeline` does not inherit `CogView4LoraLoaderMixin`, although the control README shows `pipe.load_lora_weights(...)`. It also retains the old strict prompt/negative embed shape check, even though CogView4 runs cond/uncond in separate transformer calls and the base pipeline was already relaxed. Finally, the control denoising loop does not set `"cond"` / `"uncond"` cache contexts; that cache-context facet is already tracked generally by issue #12760.

Impact:
Control LoRA loading is unavailable, split text-encoding workflows fail for valid prompt/negative embeddings with different sequence lengths, and cache hooks can share state between conditional and unconditional passes.

Reproduction:
```python
import inspect
import torch
from diffusers import CogView4ControlPipeline

print("has load_lora_weights:", hasattr(CogView4ControlPipeline, "load_lora_weights"))

pipe = object.__new__(CogView4ControlPipeline)
pipe._callback_tensor_inputs = ["latents", "prompt_embeds", "negative_prompt_embeds"]
try:
pipe.check_inputs(
prompt=None,
height=16,
width=16,
negative_prompt=None,
callback_on_step_end_tensor_inputs=["latents"],
prompt_embeds=torch.randn(1, 12, 8),
negative_prompt_embeds=torch.randn(1, 4, 8),
)
except Exception as e:
print(type(e).__name__, e)

source = inspect.getsource(CogView4ControlPipeline.__call__)
print("uses cache_context:", 'cache_context("cond")' in source and 'cache_context("uncond")' in source)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4.py#L137
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4.py#L624-L648
Related prior base-pipeline fix: https://github.com/huggingface/diffusers/pull/10966
Duplicate for cache contexts: https://github.com/huggingface/diffusers/issues/12760

Suggested fix:
```python
from ...loaders import CogView4LoraLoaderMixin

class CogView4ControlPipeline(DiffusionPipeline, CogView4LoraLoaderMixin):
...

if prompt_embeds is not None and negative_prompt_embeds is not None:
if prompt_embeds.shape[0] != negative_prompt_embeds.shape[0]:
raise ValueError(...)
if prompt_embeds.shape[-1] != negative_prompt_embeds.shape[-1]:
raise ValueError(...)

with self.transformer.cache_context("cond"):
noise_pred_cond = self.transformer(...)

with self.transformer.cache_context("uncond"):
noise_pred_uncond = self.transformer(...)
```

## Issue 6: Lazy export advertises the wrong output class

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/__init__.py#L15
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_output.py#L10-L12

Problem:
`pipelines/cogview4/__init__.py` exports `CogView4PlusPipelineOutput`, but `pipeline_output.py` defines `CogView4PipelineOutput`. The output docstring also says CogView3.

Impact:
`from diffusers.pipelines.cogview4 import CogView4PipelineOutput` fails, and autodoc/lazy import metadata is wrong.

Reproduction:
```python
from diffusers.pipelines.cogview4 import CogView4PipelineOutput
print(CogView4PipelineOutput)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview3/__init__.py#L15
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview3/pipeline_output.py#L10-L12

Suggested fix:
```python
_import_structure = {"pipeline_output": ["CogView4PipelineOutput"]}
```
Also update the output docstring to say CogView4.

## Issue 7: Control example documentation/script has runnable breakages

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/cogview4-control/README.md#L19
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/cogview4-control/README.md#L123
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/examples/cogview4-control/train_control_cogview4.py#L821-L827

Problem:
The README’s primary LoRA command references `train_control_lora_cogview4.py`, but that file is not present. The inference snippet passes `joint_attention_kwargs`, while the pipeline argument is `attention_kwargs`. The full fine-tuning script has `module.requirs_grad_(False)`, which fails when `--only_target_transformer_blocks` is used.

Impact:
Users following the example cannot launch the documented LoRA training path, can pass an unsupported inference kwarg, and can hit an AttributeError in the full fine-tuning script.

Reproduction:
```python
from pathlib import Path
import torch.nn as nn

assert Path("examples/cogview4-control/train_control_lora_cogview4.py").exists()
nn.Linear(1, 1).requirs_grad_(False)
```

Relevant precedent:
The README itself says inference is performed with `CogView4ControlPipeline`, whose signature exposes `attention_kwargs`, not `joint_attention_kwargs`.

Suggested fix:
```python
# train_control_cogview4.py
module.requires_grad_(False)
```
Add the missing LoRA script or remove/update that section, and change the README inference kwarg to:
```python
attention_kwargs={"scale": 0.9}
```

## Issue 8: Slow tests and ControlPipeline tests are missing

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/cogview4/test_cogview4.py#L32

Problem:
The CogView4 test file only defines `CogView4PipelineFastTests`. There is no `@slow` CogView4 integration test, and there is no fast or slow coverage for `CogView4ControlPipeline`.

Impact:
The control pipeline can regress independently from the base pipeline, and real checkpoint loading/inference is untested. This also leaves the slow-test requirement for the target family unmet.

Reproduction:
```python
from pathlib import Path

test_files = list(Path("tests").rglob("*cogview4*.py"))
text = "\n".join(path.read_text(encoding="utf-8") for path in test_files)

print("test files:", [str(p) for p in test_files])
print("@slow present:", "@slow" in text)
print("CogView4ControlPipeline tested:", "CogView4ControlPipeline" in text)
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/cogview3/test_cogview3plus.py#L241-L264

Suggested fix:
Add a `CogView4ControlPipelineFastTests` class with tiny synthetic components/control images, and add at least one `@slow` smoke test for the released CogView4 checkpoint family, including the control pipeline when the checkpoint is available to CI.

Start with the affected CogView4 transformer and pipeline files named in the report, then compare the QwenImage and Flux precedents linked for attention masks, backend dispatch, schedules, and callbacks. Run the supplied reproductions for each finding and inspect the control-pipeline README and training entry point. Done means the reported behaviors work consistently for base and control pipelines without breaking the documented APIs.

Vom Indexierungsmodell aus dem Issue-Text verfasst.