flux model/pipeline review

`flux` model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules.

Duplicate search status: searched current GitHub Issues and PRs for flux, affected class/function names, and failure modes. No likely duplicates found. Broader gh searches hit rate limits after targeted searches, so remaining modular-pipeline searches were checked through the GitHub connector; related PRs such as #12272 and #13482 are not duplicates.

Issue 1: Flux IP-Adapter masks are accepted but ignored

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_flux.py#L183-L268
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/transformer_flux.py#L345-L346

Problem:
FluxIPAdapterAttnProcessor.__call__ accepts ip_adapter_masks, and FluxAttention.forward explicitly quiets that kwarg, but the processor never reads or applies the masks.

Impact:
Region-specific IP-Adapter conditioning silently has no effect for Flux. Users can pass masks without warnings, making masked IP-Adapter results misleading.

Reproduction:

import torch
from diffusers.models.transformers.transformer_flux import FluxAttention, FluxIPAdapterAttnProcessor

torch.manual_seed(0)
attn = FluxAttention(
    query_dim=4,
    heads=1,
    dim_head=4,
    added_kv_proj_dim=4,
    processor=FluxIPAdapterAttnProcessor(hidden_size=4, cross_attention_dim=4, num_tokens=(2,), scale=10.0),
)

hidden = torch.randn(1, 3, 4)
encoder = torch.randn(1, 2, 4)
ip = [torch.randn(1, 2, 4)]
mask0 = [torch.zeros(1, 3, 1, 1)]
mask1 = [torch.ones(1, 3, 1, 1)]

out0 = attn(hidden, encoder_hidden_states=encoder, ip_hidden_states=ip, ip_adapter_masks=mask0)[0]
out1 = attn(hidden, encoder_hidden_states=encoder, ip_hidden_states=ip, ip_adapter_masks=mask1)[0]
print(torch.max(torch.abs(out0 - out1)).item())  # 0.0

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/attention_processor.py#L4309-L4378

Suggested fix:
Apply the same validation/downsampling pattern used by IPAdapterAttnProcessor2_0, adapted for Flux sequence-shaped hidden states. If mask support is not intended, remove ip_adapter_masks from the quieted kwargs so users get a warning.

Issue 2: `negative_prompt_embeds` do not enable true CFG in several Flux pipelines

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux_img2img.py#L902
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux_inpaint.py#L977
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux_controlnet.py#L863

Problem:
These pipelines compute true CFG only with negative_prompt is not None. They validate and accept negative_prompt_embeds / negative_pooled_prompt_embeds, but embeddings alone do not activate true CFG.

Impact:
Advanced users passing precomputed negative embeddings get silently different behavior from text negative prompts.

Reproduction:

import torch
from diffusers import FluxImg2ImgPipeline

pipe = object.__new__(FluxImg2ImgPipeline)
pipe.vae_scale_factor = 8
pipe._callback_tensor_inputs = ["latents", "prompt_embeds"]

pipe.check_inputs(
    prompt=None,
    prompt_2=None,
    strength=0.5,
    height=64,
    width=64,
    prompt_embeds=torch.zeros(2, 4, 8),
    pooled_prompt_embeds=torch.zeros(2, 8),
    negative_prompt_embeds=torch.zeros(2, 4, 8),
    negative_pooled_prompt_embeds=torch.zeros(2, 8),
    callback_on_step_end_tensor_inputs=["latents"],
    max_sequence_length=48,
)

negative_prompt = None
true_cfg_scale = 2.0
print(true_cfg_scale > 1 and negative_prompt is not None)  # False

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux.py#L820-L823
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux_kontext.py#L946-L949

Suggested fix:

has_neg_prompt = negative_prompt is not None or (
    negative_prompt_embeds is not None and negative_pooled_prompt_embeds is not None
)
do_true_cfg = true_cfg_scale > 1 and has_neg_prompt

Issue 3: Base Flux and Flux Kontext accept mismatched negative prompt embeddings

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux.py#L483-L500
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux_kontext.py#L534-L551

Problem:
FluxPipeline and FluxKontextPipeline check that negative pooled embeddings are provided with negative prompt embeddings, but they do not verify that negative and positive embedding shapes match.

Impact:
Shape or batch mismatches are accepted at input validation time and fail later, or worse, produce hard-to-debug true CFG behavior.

Reproduction:

import torch
from diffusers import FluxPipeline, FluxKontextPipeline

for cls in (FluxPipeline, FluxKontextPipeline):
    pipe = object.__new__(cls)
    pipe.vae_scale_factor = 8
    pipe._callback_tensor_inputs = ["latents", "prompt_embeds"]
    pipe.check_inputs(
        prompt=None,
        prompt_2=None,
        height=64,
        width=64,
        prompt_embeds=torch.zeros(2, 4, 8),
        pooled_prompt_embeds=torch.zeros(2, 8),
        negative_prompt_embeds=torch.zeros(1, 4, 8),
        negative_pooled_prompt_embeds=torch.zeros(1, 8),
        callback_on_step_end_tensor_inputs=["latents"],
        max_sequence_length=48,
    )
    print(cls.__name__, "accepted mismatched negative embeds")

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux_img2img.py#L550-L556
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux_kontext_inpaint.py#L598-L604

Suggested fix:

if prompt_embeds is not None and negative_prompt_embeds is not None:
    if prompt_embeds.shape != negative_prompt_embeds.shape:
        raise ValueError(
            "`negative_prompt_embeds` must have the same shape as `prompt_embeds`, "
            f"but got {negative_prompt_embeds.shape} != {prompt_embeds.shape}."
        )

Issue 4: Flux ControlNet img2img validates dimensions with wrong operator precedence

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux_controlnet_image_to_image.py#L442-L459

Problem:
The dimension check uses height % self.vae_scale_factor * 2, which is parsed as (height % self.vae_scale_factor) * 2, not height % (self.vae_scale_factor * 2).

Impact:
Invalid dimensions such as 72x72 pass validation when vae_scale_factor == 8, even though Flux latent packing expects dimensions divisible by 16.

Reproduction:

from diffusers import FluxControlNetImg2ImgPipeline

pipe = object.__new__(FluxControlNetImg2ImgPipeline)
pipe.vae_scale_factor = 8
pipe._callback_tensor_inputs = ["latents", "prompt_embeds", "control_image"]

pipe.check_inputs(
    prompt="x",
    prompt_2=None,
    strength=0.5,
    height=72,
    width=72,
    callback_on_step_end_tensor_inputs=["latents"],
    prompt_embeds=None,
    pooled_prompt_embeds=None,
    max_sequence_length=48,
)

print(72 % pipe.vae_scale_factor * 2)      # 0, current check
print(72 % (pipe.vae_scale_factor * 2))    # 8, intended check

Relevant precedent:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux_img2img.py#L489-L510
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux_controlnet.py#L473-L489

Suggested fix:

if height % (self.vae_scale_factor * 2) != 0 or width % (self.vae_scale_factor * 2) != 0:
    raise ValueError(...)

Issue 5: Flux Prior Redux misses tensor-batch and pooled-scale validation

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux_prior_redux.py#L143-L179
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux_prior_redux.py#L425-L435
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux_prior_redux.py#L475-L478

Problem:
check_inputs only checks image batch size when image is a list. Tensor image batches are skipped. It also checks prompt_embeds_scale list length but not pooled_prompt_embeds_scale.

Impact:
Mismatched image/prompt batches and scale lengths are accepted, then fail later during tensor arithmetic or produce confusing broadcasting behavior.

Reproduction:

import torch
from diffusers import FluxPriorReduxPipeline

pipe = object.__new__(FluxPriorReduxPipeline)
pipe.check_inputs(
    image=torch.zeros(2, 3, 32, 32),
    prompt=["first", "second", "third"],
    prompt_2=None,
    prompt_embeds=None,
    pooled_prompt_embeds=None,
    prompt_embeds_scale=[1.0],
    pooled_prompt_embeds_scale=[1.0, 1.0, 1.0],
)

print("accepted tensor image batch=2 with prompt batch=3 and prompt scale length=1")

Relevant precedent:
Flux text/image pipelines consistently validate paired prompt embedding inputs before denoising, for example:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/pipelines/flux/pipeline_flux.py#L483-L500

Suggested fix:
Determine image batch size for tensors as well as lists, then validate prompt batch size and both scale arguments against that batch size.

image_batch_size = image.shape[0] if isinstance(image, torch.Tensor) else len(image) if isinstance(image, list) else 1

for name, scale in {
    "prompt_embeds_scale": prompt_embeds_scale,
    "pooled_prompt_embeds_scale": pooled_prompt_embeds_scale,
}.items():
    if isinstance(scale, list) and len(scale) != image_batch_size:
        raise ValueError(f"`{name}` must have length {image_batch_size}, but got {len(scale)}.")

Issue 6: Flux modular pipelines import standard Flux pipelines and QwenImage modular helpers

Affected code:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/modular_pipelines/flux/before_denoise.py#L20
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/modular_pipelines/flux/before_denoise.py#L393
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/modular_pipelines/flux/before_denoise.py#L544-L609
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/modular_pipelines/flux/inputs.py#L18-L24
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/modular_pipelines/flux/inputs.py#L212-L274
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/modular_pipelines/flux/encoders.py#L171-L173

Problem:
The modular review rules prohibit hidden imports from classic pipelines and cross-family modular imports. Flux modular blocks import FluxPipeline for private helper methods and import QwenImage modular input helpers.

Impact:
This creates hidden coupling between modular Flux, classic Flux, and QwenImage. Refactors or lazy-loading changes in one family can break another, and modular Flux becomes harder to serialize, test, and maintain independently.

Reproduction:

from pathlib import Path

for path in Path("src/diffusers/modular_pipelines/flux").glob("*.py"):
    for line_no, line in enumerate(path.read_text().splitlines(), 1):
        if "from ...pipelines" in line or "from ..qwenimage" in line or "FluxPipeline._" in line:
            print(f"{path}:{line_no}: {line.strip()}")

Relevant precedent:
decoders.py keeps its Flux latent unpacking helper local instead of importing the classic pipeline:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/modular_pipelines/flux/decoders.py#L32-L44

Suggested fix:
Move shared latent packing, latent id, and dimension helpers into Flux modular-local helpers or a neutral utility module that both classic and modular Flux can import without creating pipeline-family coupling. Copy the Kontext resolution table into Flux modular code or move it to a neutral Flux constants module.

Issue 7: Slow coverage is missing for most Flux pipeline variants

Problem:
Fast tests exist for these variants, but slow/nightly coverage is missing for Flux img2img, inpaint, fill, control, control img2img, control inpaint, Kontext, Kontext inpaint, ControlNet img2img, ControlNet inpaint, and Flux modular pipelines.

Impact:
Several public Flux variants are not covered against real checkpoints, schedulers, tokenizer/text-encoder stacks, or pipeline loading paths. This is especially risky for Flux because many bugs only appear with real component shapes, offload behavior, attention processors, and true CFG paths.

Reproduction:

import ast
from pathlib import Path

files = sorted(Path("tests").glob("**/*flux*.py"))
for path in files:
    tree = ast.parse(path.read_text())
    slow_classes = []
    for node in tree.body:
        if isinstance(node, ast.ClassDef):
            decorators = {getattr(d, "id", getattr(d, "attr", "")) for d in node.decorator_list}
            if decorators & {"slow", "nightly"}:
                slow_classes.append(node.name)
    if path.match("tests/pipelines/flux/*") or path.match("tests/pipelines/controlnet_flux/*") or path.match("tests/modular_pipelines/flux/*"):
        print(path, slow_classes)

Relevant precedent:
Existing real-checkpoint Flux slow coverage is present for base Flux, Flux Redux, and base Flux ControlNet:
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/flux/test_pipeline_flux.py#L238-L303
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/flux/test_pipeline_flux_redux.py#L20-L22
https://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/tests/pipelines/controlnet_flux/test_controlnet_flux.py#L212-L214

Suggested fix:
Add at least one @slow or @nightly smoke test per missing public variant using a small deterministic inference path and real public checkpoint components where available. For modular Flux, add slow parity coverage against the corresponding classic Flux pipeline for text-to-image, image-to-image, and Kontext paths.

Description