huggingface / huggingface/diffusers
[Bug] QwenImageEdit produces incorrect outputs with transformers >=5 due to missing mm_token_type_ids
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
### Describe the bug
# [Bug] QwenImage pipeline does not forward mm_token_type_ids for transformers 5.x
## Describe the bug
When using QwenImageEdit with newer versions of `transformers`, the generated image output can differ from previous versions even with identical:
- model weights
- input image
- prompt
- random seed
- initial latent
The issue is caused by missing forwarding of `mm_token_type_ids` from the processor output to the Qwen2.5-VL text encoder.
## Environment
Affected combinations:
- diffusers: 0.38.0+
- transformers: 5.x
Tested pipeline:
- QwenImageEdit pipeline
## Root cause
`transformers` 5.x introduced `mm_token_type_ids` for explicit multimodal token identification in Qwen2.5-VL models.
This field is required by the model to construct correct multimodal 3D RoPE position ids.
The expected flow is:
```
processor output
input_ids
image_grid_thw
mm_token_type_ids
|
v
Qwen2.5-VL text encoder
|
v
3D RoPE position_ids
```
However, the current diffusers QwenImage pipeline only forwards:
```python
input_ids
attention_mask
pixel_values
image_grid_thw
```
and does not forward:
```python
mm_token_type_ids
```
As a result, with transformers 5.x, the model cannot determine the multimodal token ranges and falls back to the default position encoding path instead of constructing multimodal 3D RoPE.
The fallback does not raise an error because position_ids=None is a valid model input. However, for image-text generation, this changes the attention position information and leads to different outputs.
## Evidence
With transformers 5.x:
+ processor correctly generates mm_token_type_ids
+ input_ids, attention_mask, pixel_values, and image_grid_thw are identical
+ the first divergence appears during language model attention computation
+ multimodal 3D RoPE is not constructed because mm_token_type_ids is missing
After manually forwarding mm_token_type_ids:
+ 3D RoPE position ids become identical to the previous working path
+ language hidden states become identical
+ final generated image MD5 matches the baseline
## Proposed fix
Forward mm_token_type_ids when available in the QwenImage pipeline:
```python
outputs = self.text_encoder(
input_ids=model_inputs["input_ids"],
attention_mask=model_inputs["attention_mask"],
pixel_values=model_inputs.get("pixel_values"),
image_grid_thw=model_inputs.get("image_grid_thw"),
mm_token_type_ids=model_inputs.get("mm_token_type_ids"),
output_hidden_states=True,
)
```
This keeps compatibility with both:
+ older transformers versions where this argument is not required
+ newer transformers versions where multimodal RoPE depends on this field
## Additional notes
This is a compatibility issue between the updated multimodal input interface in transformers 5.x and the current diffusers QwenImage pipeline.
The fix only forwards an existing processor output field and does not change model behavior for existing supported configurations.
### Reproduction
Install the affected versions:
```bash
pip install diffusers==0.38.0
pip install transformers>=5.0.0
```
Run the following example:
```python
import torch
from diffusers import QwenImageEditPipeline
from PIL import Image
model_id = ""
pipe = QwenImageEditPipeline.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
)
pipe.to("cuda")
image = Image.open("input.png")
prompt = "Edit the image according to the instruction."
generator = torch.Generator(device="cuda").manual_seed(42)
output = pipe(
image=image,
prompt=prompt,
generator=generator,
)
output.images[0].save("output.png")
```
The issue can be reproduced by checking the processor output:
```python
model_inputs = pipe.processor(
text=[prompt],
images=[image],
padding=True,
return_tensors="pt",
)
print(model_inputs.keys())
```
With transformers >= 5.x, the processor returns:
```
input_ids
attention_mask
pixel_values
image_grid_thw
mm_token_type_ids
```
to the Qwen2.5-VL text encoder and drops
```python
mm_token_type_ids
```
As a result, Qwen2.5-VL cannot construct multimodal 3D RoPE position ids and falls back to the default position encoding path.
The missing argument can be verified by adding:
```python
mm_token_type_ids=model_inputs.get("mm_token_type_ids")
```
to the text_encoder call inside the QwenImage pipeline.
After forwarding mm_token_type_ids:
+ multimodal 3D RoPE position ids are correctly generated
+ language hidden states match the previous working configuration
+ generated image output becomes consistent with the expected result
### Logs
```shell
```
### System Info
Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.
- 🤗 Diffusers version: 0.38.0
- Platform: Linux-6.6.0-100.jd_b003.x86_64-x86_64-with-glibc2.35
- Running on Google Colab?: No
- Python version: 3.10.20
- PyTorch version (GPU?): 2.12.0+cu130 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 1.17.0
- Transformers version: 5.10.1
- Accelerate version: 1.13.0
- PEFT version: 0.19.1
- Bitsandbytes version: not installed
- Safetensors version: 0.8.0
- xFormers version: not installed
### Who can help?
_No response_
Contributor guide
Research direction
Start at the QwenImage pipeline text_encoder call described in the issue and inspect how processor output is passed to Qwen2.5-VL. Run the provided QwenImageEdit reproduction with transformers 5.x, then verify that mm_token_type_ids reaches the encoder and that the generated output matches the expected baseline. No specific source file or test path is named in the report.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100