microsoft / microsoft/onnxruntime-genai

Phi-4 multi-modal produces unusually poor results

Open
#1,383 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

**Describe the bug**

`microsoft/Phi-4-multimodal-instruct` describes the image as extremely corrupt beyond recognition when it can clearly be described.

**To Reproduce**
Adapted from the `phi4-mm.py` example:
```python
import onnxruntime_genai as og

model_path = "gpu/gpu-int4-rtn-block-32"
execution_provider = "cuda"
image = "36979.jpg"
text = "Describe the image."

config = og.Config(model_path)
config.clear_providers()
if execution_provider != "cpu":
config.append_provider(execution_provider)
model = og.Model(config)

processor = model.create_multimodal_processor()
tokenizer_stream = processor.create_stream()

images = og.Images.open(image)
prompt = "<|user|>\n"
prompt += "<|image_1|>\n"
prompt += f"{text}<|end|>\n<|assistant|>\n"
inputs = processor(prompt, images=images, audios=None)
params = og.GeneratorParams(model)
params.set_inputs(inputs)
params.set_search_options(max_length=768)

generator = og.Generator(model, params)

while not generator.is_done():
generator.generate_next_token()

new_token = generator.get_next_tokens()[0]
print(tokenizer_stream.decode(new_token), end="", flush=True)
```

**Expected behavior**

Similar to the output in the [HF space](https://huggingface.co/spaces/microsoft/phi-4-multimodal): "The image shows a group of people around a table with a green tablecloth. There are playing cards on the table, and the people are engaged in a card game. The room has a yellowish wall and a window with blinds."

**Actual behavior**

CUDA: "The image appears to be a highly distorted or corrupted photo. It contains various colors and shapes that do not correspond to a recognizable scene or object. The image lacks clarity and coherence, making it difficult to provide a detailed description."

CPU: "The image appears to be a highly distorted or corrupted photo. The central figure is a person wearing a white shirt and dark pants, standing in front of a green background. There is visible color fringing and pixelation throughout the image."

CPU can provide at least some detail, but still performs much worse than expected.

**Screenshots**

The image in question is from the Flickr30k dataset (zipped to avoid compression):

[36979.jpg.zip](https://github.com/user-attachments/files/19639340/36979.jpg.zip)

**Desktop (please complete the following information):**
- OS: Arch
- GPU: RTX 4090
- onnxruntime-genai-cuda: 0.7.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the adapted phi4-mm.py example and reproduce the reported outputs using the supplied Flickr30k image, first on CPU and then with CUDA. Compare the multimodal processor and provider paths, using the expected Hugging Face description as the quality reference. Done means the cause of the degraded descriptions is identified and the issue is corrected or clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.