cogview3 model/pipeline review

# `cogview3` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

Duplicate search: checked GitHub Issues/PRs for `cogview3`, affected class/output names, prompt-embed batch failures, size validation, attention backend behavior, `pooled_projection_dim`, stale checkpoint IDs, and fp16 black-image failures. Related but not full duplicates: PR https://github.com/huggingface/diffusers/pull/10211 fixed only the pipeline example checkpoint ID; issue https://github.com/huggingface/diffusers/issues/10343 covers CogView3 fp16 black images.

Local execution note: targeted reproductions were run with `.venv`. Full fast test collection did not reach target code because this environment's Torch build lacks `torch._C._distributed_c10d`.

## Issue 1: Lazy export points to a non-existent output class

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview3/__init__.py#L15
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview3/pipeline_output.py#L9-L20

Problem:
`cogview3.__init__` exports `CogView3PlusPipelineOutput`, but `pipeline_output.py` defines `CogView3PipelineOutput`. Importing the advertised lazy symbol fails.

Impact:
Public lazy imports are broken for the output type, and docs/export tooling can drift from the actual class.

Reproduction:
```python
try:
from diffusers.pipelines.cogview3 import CogView3PlusPipelineOutput
except Exception as e:
print(type(e).__name__, e)

from diffusers.pipelines.cogview3.pipeline_output import CogView3PipelineOutput
print(CogView3PipelineOutput.__name__)
```

Relevant precedent:
Docs already autodoc the real class name at:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/docs/source/en/api/pipelines/cogview3.md#L35-L37

Suggested fix:
```python
_import_structure = {"pipeline_output": ["CogView3PipelineOutput"]}

# in TYPE_CHECKING branch
from .pipeline_output import CogView3PipelineOutput
```

## Issue 2: Precomputed prompt embeddings are not expanded or dtype/device-normalized

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview3/pipeline_cogview3plus.py#L258-L292
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview3/pipeline_cogview3plus.py#L576-L631

Problem:
When `prompt_embeds` or `negative_prompt_embeds` are passed directly, CogView3 does not repeat them for `num_images_per_prompt` and does not cast/move them to the execution dtype/device. The latent batch is expanded, but the prompt batch is not.

Impact:
Valid diffusers usage with precomputed embeddings fails with batch mismatch, device mismatch, or dtype mismatch.

Reproduction:
```python
import torch
from diffusers import CogVideoXDDIMScheduler, CogView3PlusPipeline, CogView3PlusTransformer2DModel

def make_pipe(dtype=torch.float32):
transformer = CogView3PlusTransformer2DModel(
patch_size=2, in_channels=4, num_layers=1, attention_head_dim=4,
num_attention_heads=2, out_channels=4, text_embed_dim=8,
time_embed_dim=8, condition_dim=2, pos_embed_max_size=8, sample_size=2,
).to(dtype=dtype)
pipe = CogView3PlusPipeline(None, None, None, transformer, CogVideoXDDIMScheduler())
pipe.set_progress_bar_config(disable=True)
return pipe

try:
make_pipe()(prompt_embeds=torch.randn(1, 8, 8), num_images_per_prompt=2,
num_inference_steps=1, guidance_scale=1.0, height=16, width=16, output_type="latent")
except Exception as e:
print(type(e).__name__, e)

try:
make_pipe(torch.float16)(prompt_embeds=torch.randn(1, 8, 8, dtype=torch.float32),
num_inference_steps=1, guidance_scale=1.0,
height=16, width=16, output_type="latent")
except Exception as e:
print(type(e).__name__, e)
```

Relevant precedent:
CogView4 expands precomputed embeds:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4.py#L269-L296

Suggested fix:
```python
if prompt_embeds is None:
prompt_embeds = self._get_t5_prompt_embeds(...)
else:
dtype = dtype or self.transformer.dtype
prompt_embeds = prompt_embeds.to(device=device, dtype=dtype)
bs_embed, seq_len, _ = prompt_embeds.shape
prompt_embeds = prompt_embeds.repeat(1, num_images_per_prompt, 1)
prompt_embeds = prompt_embeds.view(bs_embed * num_images_per_prompt, seq_len, -1)

if do_classifier_free_guidance and negative_prompt_embeds is None and negative_prompt is None:
negative_prompt_embeds = prompt_embeds.new_zeros(prompt_embeds.shape)
elif do_classifier_free_guidance and negative_prompt_embeds is not None:
negative_prompt_embeds = negative_prompt_embeds.to(device=device, dtype=prompt_embeds.dtype)
bs_embed, seq_len, _ = negative_prompt_embeds.shape
negative_prompt_embeds = negative_prompt_embeds.repeat(1, num_images_per_prompt, 1)
negative_prompt_embeds = negative_prompt_embeds.view(bs_embed * num_images_per_prompt, seq_len, -1)
```

## Issue 3: Height/width validation ignores transformer patch size

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview3/pipeline_cogview3plus.py#L336-L348

Problem:
The pipeline only requires dimensions divisible by `8`, but CogView3 patchifies VAE latents with `patch_size=2`. A `24x24` image passes `check_inputs`, then fails inside the transformer because latent size `3x3` is not divisible by patch size.

Impact:
Users get a late internal transformer error instead of a clear pipeline validation error.

Reproduction:
```python
import torch
from diffusers import CogVideoXDDIMScheduler, CogView3PlusPipeline, CogView3PlusTransformer2DModel

transformer = CogView3PlusTransformer2DModel(
patch_size=2, in_channels=4, num_layers=1, attention_head_dim=4,
num_attention_heads=2, out_channels=4, text_embed_dim=8,
time_embed_dim=8, condition_dim=2, pos_embed_max_size=8, sample_size=2,
)
pipe = CogView3PlusPipeline(None, None, None, transformer, CogVideoXDDIMScheduler())
pipe.set_progress_bar_config(disable=True)

try:
pipe(prompt_embeds=torch.randn(1, 8, 8), num_inference_steps=1,
guidance_scale=1.0, height=24, width=24, output_type="latent")
except Exception as e:
print(type(e).__name__, e)
```

Relevant precedent:
CogView4 validates `16`, matching `vae_scale_factor * patch_size`:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/cogview4/pipeline_cogview4.py#L328-L329

Suggested fix:
```python
divisibility = self.vae_scale_factor * self.transformer.config.patch_size
if height % divisibility != 0 or width % divisibility != 0:
raise ValueError(
f"`height` and `width` have to be divisible by {divisibility} but are {height} and {width}."
)
```

## Issue 4: Attention backend selection is effectively ignored

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_cogview3plus.py#L21-L22
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_cogview3plus.py#L58-L68
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/attention_processor.py#L2277-L2331

Problem:
CogView3Plus reuses `CogVideoXAttnProcessor2_0`, which calls `F.scaled_dot_product_attention` directly and has no `_attention_backend`. `model.set_attention_backend(...)` skips it, so backend dispatch support is a no-op for this transformer.

Impact:
Users cannot reliably select supported attention backends for CogView3Plus, and the implementation violates the review rule requiring model-local processors to route through `dispatch_attention_fn`.

Reproduction:
```python
from diffusers import CogView3PlusTransformer2DModel

model = CogView3PlusTransformer2DModel(
patch_size=2, in_channels=4, num_layers=1, attention_head_dim=4,
num_attention_heads=2, out_channels=4, text_embed_dim=8,
time_embed_dim=8, condition_dim=2, pos_embed_max_size=8, sample_size=2,
)
name, processor = next(iter(model.attn_processors.items()))
print(name, processor.__class__.__name__, hasattr(processor, "_attention_backend"))
model.set_attention_backend("native")
print(hasattr(processor, "_attention_backend"), getattr(processor, "_attention_backend", None))
```

Relevant precedent:
QwenImage routes attention through `dispatch_attention_fn`:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_qwenimage.py#L562-L569

Suggested fix:
Implement a CogView3Plus-local attention processor, with `_attention_backend` and `_parallel_config`, and call `dispatch_attention_fn(...)` instead of `F.scaled_dot_product_attention(...)`. If doing the full rule-compliant refactor, also define a local attention module inheriting `AttentionModuleMixin`.

## Issue 5: Released transformer config contains an ignored key on load

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_cogview3plus.py#L168-L199

Problem:
The published `THUDM/CogView3-Plus-3B` transformer config includes `pooled_projection_dim`, but the model constructor does not accept it. Loading emits an unexpected-config warning even though the value matches the derived value.

Impact:
Users see a spurious load warning for the official checkpoint, and config compatibility remains noisy.

Reproduction:
```python
from diffusers import CogView3PlusTransformer2DModel

config = {
"patch_size": 2, "in_channels": 4, "num_layers": 1,
"attention_head_dim": 4, "num_attention_heads": 2,
"out_channels": 4, "text_embed_dim": 8, "time_embed_dim": 8,
"condition_dim": 2, "pos_embed_max_size": 8, "sample_size": 2,
"pooled_projection_dim": 12,
}
model = CogView3PlusTransformer2DModel.from_config(config)
print(model.pooled_projection_dim)
```

Relevant precedent:
Other transformer configs expose `pooled_projection_dim` directly where it is serialized:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/hunyuan_transformer_2d.py#L264-L294

Suggested fix:
```python
def __init__(..., pooled_projection_dim: int | None = None, ...):
...
self.pooled_projection_dim = pooled_projection_dim or 3 * 2 * condition_dim
self.register_to_config(pooled_projection_dim=self.pooled_projection_dim)
```

## Issue 6: CogView3 tests/docs do not provide meaningful coverage

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/cogview3/test_cogview3plus.py#L135-L138
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/cogview3/test_cogview3plus.py#L256-L276
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/docs/source/en/api/models/cogview3plus_transformer2d.md#L19-L21

Problem:
Fast tests compare against `torch.randn(...)` with `1e10` tolerance, so they only check shape. The slow test uses non-existent model ID `THUDM/CogView3Plus-3b`, loads unsupported `torch.float16`, and compares `images[0]` to a random expected array with an incompatible leading batch dimension. The model docs also use the stale checkpoint ID.

Impact:
Fast and slow tests exist, but they do not catch output regressions. The slow test cannot validate the official checkpoint as written.

Reproduction:
```python
from huggingface_hub import model_info
import numpy as np
from numpy.linalg import norm

for model_id in ["THUDM/CogView3Plus-3b", "THUDM/CogView3-Plus-3B"]:
try:
print(model_id, model_info(model_id).id)
except Exception as e:
print(model_id, type(e).__name__)

def numpy_cosine_similarity_distance(a, b):
similarity = np.dot(a, b) / (norm(a) * norm(b))
return 1.0 - similarity.mean()

try:
image = np.zeros((4, 4, 3), dtype=np.float32)
expected_image = np.random.randn(1, 4, 4, 3).astype(np.float32)
print(numpy_cosine_similarity_distance(image, expected_image))
except Exception as e:
print(type(e).__name__, e)
```

Relevant precedent:
PR fixing the pipeline example checkpoint ID:
https://github.com/huggingface/diffusers/pull/10211

Existing fp16 black-image duplicate:
https://github.com/huggingface/diffusers/issues/10343

Suggested fix:
Update docs/tests to `THUDM/CogView3-Plus-3B`, use `torch.bfloat16` for slow inference, and replace random expectations with deterministic expected slices or stored expected statistics generated from the official checkpoint. Fast `test_inference` should assert a small image slice with a real tolerance, not `1e10`.

Start with the affected CogView3 files in src/diffusers/pipelines/cogview3, src/diffusers/models/transformers/transformer_cogview3plus.py, and attention_processor.py, then run the issue's targeted reproductions. Review tests/pipelines/cogview3/test_cogview3plus.py and the linked model and pipeline docs. Done means the reported import, embedding, validation, attention, config, test, and checkpoint-reference problems are covered without regressions.

Vom Indexierungsmodell aus dem Issue-Text verfasst.