microsoft / microsoft/onnxruntime-genai
Feature request: add ONNX Runtime GenAI support for Granite Speech models (e.g. granite-4.0-1b-speech-ONNX)
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
### Summary
Please consider adding ONNX Runtime GenAI support for the Granite Speech model family, for example:
- https://huggingface.co/onnx-community/granite-4.0-1b-speech-ONNX
My goal is to use this model from native .NET/C# through ONNX Runtime GenAI, without having to manually orchestrate
the speech/audio pipeline outside the runtime.
### Why I am filing this here
This looks like a GenAI model-family support gap rather than a core ONNX Runtime issue.
The model is already published as ONNX and includes the expected tokenizer / processor config files on Hugging
Face, but it does not appear to be a drop-in ORT GenAI model today.
From looking through the current ORT GenAI source/docs:
- `whisper` appears to be the supported audio-language model path
- `phi4mm` appears to be the supported multimodal audio-capable path
- Granite Speech / `granite_speech` does not appear to be a registered model family
So this seems like the right repository for a feature request.
### Model details
The Granite Speech ONNX export is split into multiple ONNX graphs, for example:
- `audio_encoder`
- `embed_tokens`
- `decoder_model_merged`
along with external `*.onnx_data` shards and processor/tokenizer configs.
Relevant files on the model repo include:
- `config.json`
- `processor_config.json`
- `preprocessor_config.json`
- `tokenizer.json`
- `tokenizer_config.json`
### What I would like to see
Support for Granite Speech models in ORT GenAI such that a model like:
- `onnx-community/granite-4.0-1b-speech-ONNX`
can be used through the normal ORT GenAI flow, ideally including:
1. a supported model type / processor path for Granite Speech
2. guidance or tooling for generating a valid `genai_config.json`
3. support for the speech + text embedding / decoding path without requiring users to manually reproduce the
orchestration logic themselves
4. usability from .NET/C# bindings, not only Python
### Current limitation
Today, using this model from C# appears to require substantial custom orchestration with plain ONNX Runtime,
including things like:
- audio preprocessing / feature extraction
- managing the split ONNX graphs
- reproducing the audio/text embedding flow
- handling generation state / KV cache
That makes the model technically usable, but not really practical in the same way as currently supported ORT GenAI
model families.
### Why this would be useful
Granite Speech is a compact speech model published in ONNX form and seems like a good fit for local/on-device use
cases.
Having first-class ORT GenAI support would make it much easier to use from:
- .NET / C#
- Windows desktop apps
- local/offline transcription scenarios
### Request
Is Granite Speech support something you would consider adding?
If not currently planned, would you accept a contribution / PR for this model family?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the existing whisper and phi4mm model paths in the ORT GenAI source and documentation, then inspect the Granite Speech model files such as config.json, processor_config.json, and the split ONNX graphs. A complete implementation would register the Granite Speech model type, support its audio/text embedding and decoding flow, provide genai_config.json guidance, and work through the .NET/C# bindings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, csharp
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100