huggingface / huggingface/optimum-intel

[OpenVINO] Support Speculative Decoding in Gemma 4 with model type 'gemma4_assistant' and 'gemma4_unified_assistant'

Open
#1,878 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
620
Forks
270
Avg merge
3d 10h
Merged PRs (30d)
22

Description

Hi. I plan to use the assistant model of Gemma 4 for Speculative Decoding (i.e. Multi-token Prediction). Unfortunately, it seems to be not supported here yet. The issue is similar to https://github.com/huggingface/optimum-intel/issues/1764, which has been resolved for 12B model (right?). I found that @openarcbob also mentioned the export for the assistant model there, but the topic seemed to be forgotten or overlooked, and the issue became focusing on 12B model in general instead. That's why I separate the topic to here instead.

Here is the packages' version:
- openvino 2026.2.1
- openvino-tokenizers 2026.2.1.0
- optimum 2.2.0.dev0
- optimum-intel 2.1.0.dev0+74f530e
- transformers 5.5.4

And I got the same error as in the source issue:
```
optimum-cli export openvino --model google/gemma-4-12B-it-qat-q4_0-unquantized-assistant --task image-text-to-text gemma-4-12B-it-qat-q4_0-unquantized-assistant
```
> ValueError: The checkpoint you are trying to load has model type gemma4_unified_assistant but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

Then I updated transformers to recognize such model type:
```
# 5.10 is needed for 'gemma4_unified[_asisstant]' model type (or just 5.8.0 for 'gemma4_assistant' model type)
pip install transformers==5.10
```
> ValueError: Trying to export a gemma4_unified_assistant model, that is a custom or unsupported architecture, but no custom export configuration was passed as custom_export_configs. Please refer to https://huggingface.co/docs/optimum/main/en/exporters/onnx/usage_guides/export_a_model#custom-export-of-transformers-models for an example on how to export custom models. Please open an issue at https://github.com/huggingface/optimum-intel/issues if you would like the model type gemma4_unified_assistant to be supported natively in the OpenVINO export.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the optimum-cli export openvino entry point and trace how model types are matched to export configurations. Reproduce the command with both gemma4_assistant and gemma4_unified_assistant, then verify that the assistant models export without requiring custom_export_configs and can be used for speculative decoding.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.