microsoft / microsoft/onnxruntime-genai
Phi-4 multi-modal produces unusually poor results
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
**Describe the bug**
`microsoft/Phi-4-multimodal-instruct` describes the image as extremely corrupt beyond recognition when it can clearly be described.
**To Reproduce**
Adapted from the `phi4-mm.py` example:
```python
import onnxruntime_genai as og
model_path = "gpu/gpu-int4-rtn-block-32"
execution_provider = "cuda"
image = "36979.jpg"
text = "Describe the image."
config = og.Config(model_path)
config.clear_providers()
if execution_provider != "cpu":
config.append_provider(execution_provider)
model = og.Model(config)
processor = model.create_multimodal_processor()
tokenizer_stream = processor.create_stream()
images = og.Images.open(image)
prompt = "<|user|>\n"
prompt += "<|image_1|>\n"
prompt += f"{text}<|end|>\n<|assistant|>\n"
inputs = processor(prompt, images=images, audios=None)
params = og.GeneratorParams(model)
params.set_inputs(inputs)
params.set_search_options(max_length=768)
generator = og.Generator(model, params)
while not generator.is_done():
generator.generate_next_token()
new_token = generator.get_next_tokens()[0]
print(tokenizer_stream.decode(new_token), end="", flush=True)
```
**Expected behavior**
Similar to the output in the [HF space](https://huggingface.co/spaces/microsoft/phi-4-multimodal): "The image shows a group of people around a table with a green tablecloth. There are playing cards on the table, and the people are engaged in a card game. The room has a yellowish wall and a window with blinds."
**Actual behavior**
CUDA: "The image appears to be a highly distorted or corrupted photo. It contains various colors and shapes that do not correspond to a recognizable scene or object. The image lacks clarity and coherence, making it difficult to provide a detailed description."
CPU: "The image appears to be a highly distorted or corrupted photo. The central figure is a person wearing a white shirt and dark pants, standing in front of a green background. There is visible color fringing and pixelation throughout the image."
CPU can provide at least some detail, but still performs much worse than expected.
**Screenshots**
The image in question is from the Flickr30k dataset (zipped to avoid compression):
[36979.jpg.zip](https://github.com/user-attachments/files/19639340/36979.jpg.zip)
**Desktop (please complete the following information):**
- OS: Arch
- GPU: RTX 4090
- onnxruntime-genai-cuda: 0.7.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the adapted phi4-mm.py example and reproduce the reported outputs using the supplied Flickr30k image, first on CPU and then with CUDA. Compare the multimodal processor and provider paths, using the expected Hugging Face description as the quality reference. Done means the cause of the degraded descriptions is identified and the issue is corrected or clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100