consisid model/pipeline review

# `consisid` model/pipeline review

Commit tested: `0f1abc4ae8b0eb2a3b40e82a310507281144c423`

Review performed against the repository review rules.

Duplicate search performed with `gh search issues/prs` for `ConsisID`, `ConsisIDPipeline`, `ConsisIDTransformer3DModel`, `num_videos_per_prompt`, `id_cond`, `latents dtype`, `_no_split_modules`, `attention backend`, and `prepare_face_models CUDAExecutionProvider`. I found no direct duplicate for the items below. Closed issue https://github.com/huggingface/diffusers/issues/10659 contains a related ONNX provider warning in logs, but it does not track the provider/device bug directly.

## Issue 1: `num_videos_per_prompt` is silently ignored

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/consisid/pipeline_consisid.py#L674
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/consisid/pipeline_consisid.py#L793

Problem:
`__call__` exposes `num_videos_per_prompt`, but resets it to `1` before prompt encoding and latent preparation. Users requesting multiple videos per prompt get one output without an error.

Impact:
The public API lies about batch semantics and downstream tests do not catch it.

Reproduction:
```python
import torch
from PIL import Image
from transformers import AutoConfig, AutoTokenizer, T5EncoderModel
from diffusers import AutoencoderKLCogVideoX, ConsisIDPipeline, ConsisIDTransformer3DModel, DDIMScheduler

transformer = ConsisIDTransformer3DModel(
num_attention_heads=2, attention_head_dim=16, in_channels=8, out_channels=4,
time_embed_dim=2, text_embed_dim=32, num_layers=1, sample_width=2,
sample_height=2, sample_frames=9, patch_size=2, temporal_compression_ratio=4,
max_text_seq_length=16, use_rotary_positional_embeddings=True,
use_learned_positional_embeddings=True, is_train_face=False,
)
vae = AutoencoderKLCogVideoX(
in_channels=3, out_channels=3, down_block_types=("CogVideoXDownBlock3D",) * 4,
up_block_types=("CogVideoXUpBlock3D",) * 4, block_out_channels=(8, 8, 8, 8),
latent_channels=4, layers_per_block=1, norm_num_groups=2, temporal_compression_ratio=4,
)
config = AutoConfig.from_pretrained("hf-internal-testing/tiny-random-t5")
pipe = ConsisIDPipeline(
AutoTokenizer.from_pretrained("hf-internal-testing/tiny-random-t5"),
T5EncoderModel(config), vae, transformer, DDIMScheduler(),
)
pipe.set_progress_bar_config(disable=True)
frames = pipe(
image=Image.new("RGB", (16, 16)), prompt="dance monkey", negative_prompt="",
generator=torch.Generator(device="cpu").manual_seed(0), num_inference_steps=1,
guidance_scale=1.0, height=16, width=16, num_frames=8, max_sequence_length=16,
num_videos_per_prompt=2, output_type="pt",
).frames
print(frames.shape) # torch.Size([1, 8, 3, 16, 16]), expected batch 2
```

Relevant precedent:
`WanPipeline` keeps and uses `num_videos_per_prompt` through prompt expansion and latent batch sizing:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/wan/pipeline_wan.py#L537-L560

Suggested fix:
Support the argument instead of resetting it. Repeat image latents and identity tensors per prompt, or reject unsupported values explicitly:
```python
if num_videos_per_prompt != 1:
raise ValueError("`num_videos_per_prompt > 1` is not currently supported by ConsisIDPipeline.")
```

## Issue 2: Identity tensors are not CFG/batch expanded

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/consisid/pipeline_consisid.py#L893-L912
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/consisid_transformer_3d.py#L637-L645
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/consisid_transformer_3d.py#L687-L691

Problem:
CFG doubles `latents`, `image_latents`, and `prompt_embeds`, but forwards `id_cond` and `id_vit_hidden` unchanged. Batched per-sample identity embeddings then fail in the facial cross-attention path.

Impact:
Batched ConsisID generation with per-image identities crashes under normal CFG settings.

Reproduction:
```python
import torch
from diffusers import ConsisIDTransformer3DModel

model = ConsisIDTransformer3DModel(
num_attention_heads=2, attention_head_dim=8, in_channels=4, out_channels=4,
time_embed_dim=2, text_embed_dim=8, num_layers=1, sample_width=8, sample_height=8,
sample_frames=8, patch_size=2, temporal_compression_ratio=4, max_text_seq_length=8,
cross_attn_interval=1, is_train_face=True, cross_attn_dim_head=1, cross_attn_num_heads=1,
LFE_id_dim=2, LFE_vit_dim=2, LFE_depth=5, LFE_dim_head=8, LFE_num_heads=2,
LFE_num_id_token=1, LFE_num_querie=1, LFE_output_dim=10, LFE_ff_mult=1, LFE_num_scale=1,
)
model(
hidden_states=torch.randn(4, 1, 4, 8, 8), # CFG-expanded batch
encoder_hidden_states=torch.randn(4, 8, 8),
timestep=torch.arange(4),
id_cond=torch.ones(2, 2), # original prompt batch
id_vit_hidden=[torch.ones(2, 2, 2)],
)
```

Relevant precedent:
Pipelines that duplicate conditional inputs for CFG keep all denoiser inputs aligned before calling the transformer.

Suggested fix:
Validate and expand identity inputs in the pipeline before the denoising loop:
```python
def expand_face_tensor(tensor):
if tensor.shape[0] == 1 and batch_size > 1:
tensor = tensor.expand(batch_size, *tensor.shape[1:])
tensor = tensor.repeat_interleave(num_videos_per_prompt, dim=0)
if do_classifier_free_guidance:
tensor = torch.cat([tensor, tensor], dim=0)
return tensor.to(device=device, dtype=prompt_embeds.dtype)
```

## Issue 3: Provided latents are not cast to the pipeline dtype

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/consisid/pipeline_consisid.py#L512-L518

Problem:
When users pass `latents`, `prepare_latents` only moves them to `device`, not `dtype`. In a bf16 pipeline, fp32 latents promote the concatenated transformer input to fp32 and then hit bf16 weights.

Impact:
Common reproducibility workflows using pre-generated fp32 latents fail in mixed/bfloat16 inference.

Reproduction:
```python
# Same tiny pipe setup as Issue 1, then:
pipe = pipe.to(dtype=torch.bfloat16)
pipe(
image=Image.new("RGB", (16, 16)), prompt="dance monkey", negative_prompt="",
generator=torch.Generator(device="cpu").manual_seed(0),
latents=torch.randn(1, 2, 4, 2, 2), # fp32
num_inference_steps=1, guidance_scale=1.0, height=16, width=16,
num_frames=8, max_sequence_length=16, output_type="pt",
)
# RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16
```

Relevant precedent:
Most pipeline latent preparation paths cast supplied latents to the requested execution dtype.

Suggested fix:
```python
latents = latents.to(device=device, dtype=dtype)
```

## Issue 4: `device_map="auto"` is unsupported because `_no_split_modules` is missing

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/consisid_transformer_3d.py#L351-L460

Problem:
`ConsisIDTransformer3DModel` inherits `ModelMixin`, but does not define `_no_split_modules`. The shared loader rejects automatic device maps for such models.

Impact:
Users cannot use `device_map="auto"`/balanced placement for a large video transformer that needs memory-aware loading.

Reproduction:
```python
from diffusers import ConsisIDTransformer3DModel

model = ConsisIDTransformer3DModel(
num_attention_heads=2, attention_head_dim=8, in_channels=4, out_channels=4,
time_embed_dim=2, text_embed_dim=8, num_layers=1, sample_width=8, sample_height=8,
sample_frames=8, patch_size=2, temporal_compression_ratio=4, max_text_seq_length=8,
cross_attn_interval=1, is_train_face=False,
)
print(model._get_no_split_modules("auto"))
# ValueError: ConsisIDTransformer3DModel does not support `device_map='auto'`.
```

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/cogvideox_transformer_3d.py#L217-L218

Suggested fix:
```python
_no_split_modules = ["ConsisIDBlock", "CogVideoXPatchEmbed", "LocalFacialExtractor", "PerceiverCrossAttention"]
```

## Issue 5: Attention backend selection cannot affect ConsisID attention

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/consisid_transformer_3d.py#L25-L28
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/consisid_transformer_3d.py#L289-L298
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/attention_processor.py#L2277-L2330

Problem:
ConsisID uses `CogVideoXAttnProcessor2_0`, which calls `F.scaled_dot_product_attention` directly and has no `_attention_backend` field. The review rules require model-local attention processors using `dispatch_attention_fn`.

Impact:
`model.set_attention_backend(...)` cannot route ConsisID attention through configured backends or context-parallel-aware dispatch.

Reproduction:
```python
from diffusers import ConsisIDTransformer3DModel

model = ConsisIDTransformer3DModel(
num_attention_heads=2, attention_head_dim=8, in_channels=4, out_channels=4,
time_embed_dim=2, text_embed_dim=8, num_layers=1, sample_width=8, sample_height=8,
sample_frames=8, patch_size=2, temporal_compression_ratio=4, max_text_seq_length=8,
cross_attn_interval=1, is_train_face=False,
)
processor = next(iter(model.attn_processors.values()))
print(processor.__class__.__name__, hasattr(processor, "_attention_backend"))
# CogVideoXAttnProcessor2_0 False
```

Relevant precedent:
`transformer_wan.py` and `transformer_flux.py` define model-local processors that call `dispatch_attention_fn`.

Suggested fix:
Port the ConsisID/CogVideoX joint attention processor into `consisid_transformer_3d.py` and call `dispatch_attention_fn(..., backend=self._attention_backend, parallel_config=self._parallel_config)`.

## Issue 6: `prepare_face_models` hard-codes CUDA ONNX providers

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/consisid/consisid_utils.py#L294-L350

Problem:
The helper accepts `device`, documents CPU/XPU-style values, but always creates InsightFace ONNX models with `providers=["CUDAExecutionProvider"]` and `ctx_id=0`.

Impact:
CPU-only or non-CUDA users get provider warnings/fallbacks or incorrect device setup from a public helper that appears device-aware.

Reproduction:
```python
from pathlib import Path

source = Path("src/diffusers/pipelines/consisid/consisid_utils.py").read_text()
for needle in ['providers=["CUDAExecutionProvider"]', "prepare(ctx_id=0)"]:
print(needle, needle in source)
```

Relevant precedent:
Diffusers device helpers usually derive provider/device behavior from the requested execution device instead of hard-coding CUDA.

Suggested fix:
```python
device_type = torch.device(device).type
providers = ["CUDAExecutionProvider", "CPUExecutionProvider"] if device_type == "cuda" else ["CPUExecutionProvider"]
ctx_id = 0 if device_type == "cuda" else -1
```

## Issue 7: Tests exist but key assertions are placeholders

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/consisid/test_consisid.py#L179-L181
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/consisid/test_consisid.py#L363-L366

Problem:
The fast test compares output to random noise with a `1e10` threshold. The slow test also compares against freshly generated random noise instead of a fixed expected slice. Slow tests are present, but they do not validate model/pipeline correctness.

Impact:
The runtime bugs above are not caught, and slow coverage can pass or fail for reasons unrelated to ConsisID behavior.

Reproduction:
```python
from pathlib import Path

text = Path("tests/pipelines/consisid/test_consisid.py").read_text()
print("1e10" in text)
print("expected_video = torch.randn" in text)
```

Relevant precedent:
Other pipeline slow tests pin deterministic expected output slices or cosine distances against fixed fixtures.

Suggested fix:
Replace placeholder assertions with stable expected slices, and add fast tests for:
```python
# num_videos_per_prompt=2 returns batch 2 or raises a clear ValueError
# batched id_cond/id_vit_hidden with guidance_scale > 1
# fp32 user latents in a bf16 pipeline
# ConsisIDTransformer3DModel._get_no_split_modules("auto")
```

Test execution note: I attempted `./.venv/Scripts/python.exe -m pytest tests/models/transformers/test_models_transformer_consisid.py tests/pipelines/consisid/test_consisid.py -q -m "not slow"`, but collection failed in this local `.venv` because the installed Windows PyTorch build lacks `torch._C._distributed_c10d`, imported through shared test mixins.

Start with the affected entry points in src/diffusers/pipelines/consisid/pipeline_consisid.py, consisid_utils.py, and src/diffusers/models/transformers/consisid_transformer_3d.py, then review the related attention processor and tests/pipelines/consisid/test_consisid.py. Run the supplied reproductions and existing ConsisID tests first. Done means each reported batch, dtype, device-map, attention-backend, provider, and assertion behavior is covered by working fixes and meaningful tests.

Written by the indexing model from the issue text.