microsoft / microsoft/onnxruntime-genai
Phi-4 audio task prompts don't work at all (Apple, onnx, dotnet)
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
**Describe the bug**
`Phi-4-multimodal-onnx` audio task prompt `<|user|><|audio_1|>Transcribe the audio clip into text.<|end|><|assistant|>` responses with this instead of the text transcription:
```
I'm sorry, but I can't watch or listen to an attached audio file. If you can provide the spoken content in text form, I can help transcribe it for you.
```
But audio query prompt `<|user|><|audio_1|><|end|><|assistant|>` responses as expected:
```
As an AI, I don't have real-time capabilities to check current weather conditions. I would recommend checking a reliable weather source such as weather.com or your preferred weather application for the most accurate and current information.
```
**To Reproduce**
Steps to reproduce the behaviour:
1. Build onnx model for cpu as per [guide](https://github.com/microsoft/onnxruntime-genai/blob/main/examples/python/phi-4-multi-modal.md)
2. `dotnet run` the sample app
**Expected behaviour**
Audio task prompt `<|user|><|audio_1|>Transcribe the audio clip into text.<|end|><|assistant|>` should return same as py sample app:
```
What's the weather like in San Francisco right now?
```
**Desktop (please complete the following information):**
Apple M1 Max 64Gb @ Sequoia 15.4.1
**Additional context (building model)**
1. Building `onnxruntime-genai`:
```
git clone https://github.com/microsoft/onnxruntime-genai
cd onnxruntime-genai
python build.py --config Release
dotnet build --configuration Release
```
2. Installing dependencies:
```
pip install backoff numpy==1.26.4 torch==2.6.0 torchaudio==2.6.0 torchvision onnx onnxscript peft requests scipy soundfile transformers
pip install ../onnxruntime-genai/build/macOS/Release/wheel/*.whl
pip uninstall onnxruntime
pip install --pre --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ onnxruntime
```
3. Modifying `builder.py`:
```
--onnxruntime.quantization.matmul_4bits_quantizer
++onnxruntime.quantization.matmul_nbits_quantizer
```
4. Converting:
```
python3 builder.py --input ./ --output ./cpu --precision fp32 --execution_provider cpu
```
5. Copying genai_config.json, speech_processor.json and vision_processor.json files from [gpu/gpu-int4-rtn-block-32](https://huggingface.co/microsoft/Phi-4-multimodal-instruct-onnx/tree/main/gpu).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked Phi-4 multimodal guide, then reproduce the behavior in the .NET sample and compare it with the Python sample. Inspect builder.py and the copied genai_config.json, speech_processor.json, and vision_processor.json files to identify where audio task prompts diverge. Done means the .NET sample returns the expected transcription for the task prompt.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- ai-infra-agents, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100