microsoft / microsoft/onnxruntime-genai

Feature request: add ONNX Runtime GenAI support for Granite Speech models (e.g. granite-4.0-1b-speech-ONNX)

Open
#2,072 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

### Summary

Please consider adding ONNX Runtime GenAI support for the Granite Speech model family, for example:

- https://huggingface.co/onnx-community/granite-4.0-1b-speech-ONNX

My goal is to use this model from native .NET/C# through ONNX Runtime GenAI, without having to manually orchestrate
the speech/audio pipeline outside the runtime.

### Why I am filing this here

This looks like a GenAI model-family support gap rather than a core ONNX Runtime issue.

The model is already published as ONNX and includes the expected tokenizer / processor config files on Hugging
Face, but it does not appear to be a drop-in ORT GenAI model today.

From looking through the current ORT GenAI source/docs:

- `whisper` appears to be the supported audio-language model path
- `phi4mm` appears to be the supported multimodal audio-capable path
- Granite Speech / `granite_speech` does not appear to be a registered model family

So this seems like the right repository for a feature request.

### Model details

The Granite Speech ONNX export is split into multiple ONNX graphs, for example:

- `audio_encoder`
- `embed_tokens`
- `decoder_model_merged`

along with external `*.onnx_data` shards and processor/tokenizer configs.

Relevant files on the model repo include:

- `config.json`
- `processor_config.json`
- `preprocessor_config.json`
- `tokenizer.json`
- `tokenizer_config.json`

### What I would like to see

Support for Granite Speech models in ORT GenAI such that a model like:

- `onnx-community/granite-4.0-1b-speech-ONNX`

can be used through the normal ORT GenAI flow, ideally including:

1. a supported model type / processor path for Granite Speech
2. guidance or tooling for generating a valid `genai_config.json`
3. support for the speech + text embedding / decoding path without requiring users to manually reproduce the
orchestration logic themselves
4. usability from .NET/C# bindings, not only Python

### Current limitation

Today, using this model from C# appears to require substantial custom orchestration with plain ONNX Runtime,
including things like:

- audio preprocessing / feature extraction
- managing the split ONNX graphs
- reproducing the audio/text embedding flow
- handling generation state / KV cache

That makes the model technically usable, but not really practical in the same way as currently supported ORT GenAI
model families.

### Why this would be useful

Granite Speech is a compact speech model published in ONNX form and seems like a good fit for local/on-device use
cases.

Having first-class ORT GenAI support would make it much easier to use from:

- .NET / C#
- Windows desktop apps
- local/offline transcription scenarios

### Request

Is Granite Speech support something you would consider adding?

If not currently planned, would you accept a contribution / PR for this model family?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the existing whisper and phi4mm model paths in the ORT GenAI source and documentation, then inspect the Granite Speech model files such as config.json, processor_config.json, and the split ONNX graphs. A complete implementation would register the Granite Speech model type, support its audio/text embedding and decoding flow, provide genai_config.json guidance, and work through the .NET/C# bindings.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, csharp
Domain
ai, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.