huggingface / huggingface/optimum-intel

Native OpenVINO export for Qwen3.6 / Qwen3.5 MoE (model_type=qwen3_5_moe)

Open
#1,754 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
620
Forks
270
Avg merge
3d 10h
Merged PRs (30d)
22

Description

Qwen released the Qwen3.6-35B-A3B family (registered as `model_type="qwen3_5_moe"`,
class `Qwen3_5MoeForConditionalGeneration`). As of `optimum-intel` 1.27.0,
`optimum-cli export openvino` fails before it even reaches the exporter:

```
ValueError: The checkpoint you are trying to load has model type `qwen3_5_moe`
but Transformers does not recognize this architecture.
```

with `transformers>=5.5` (which does register the architecture) the export then
fails at the OpenVINO exporter config map, which registers `qwen2`, `qwen2_moe`,
`qwen3`, `qwen3_moe`, `qwen2_vl`, `qwen2_5_vl` but not `qwen3_5_moe`.

### What makes this architecture non-trivial

Unlike `qwen3_moe`, this family is a **3:1 hybrid**: 3 Gated-DeltaNet
(linear-attention) layers per 1 full GQA-attention layer. The DeltaNet layers:

- maintain recurrent state (`conv_state` + `recurrent_state`) updated in place
inside a `Cache` object — not graph-traceable as written;
- default to CUDA-only kernels (`flash-linear-attention`, `causal-conv1d`) with
pure-torch fallbacks (`torch_recurrent_gated_delta_rule`,
`torch_causal_conv1d_update`) that *are* traceable.

A working OpenVINO export needs: (1) a model-patcher that forces the torch
fallbacks, (2) recurrent/conv state lifted to explicit model inputs/outputs
(stateful or KV-style), (3) the MoE routing expressed graph-friendly.

### Reference implementation

I have a self-contained, OV-traceable port: `QwenGatedDeltaNet`
with functional state IO, GQA attention with explicit KV cache, MoE as
compute-all+mask (single IR) **or** a backbone + per-expert split, plus a GPTQ
int4 dequantizer that reads the community checkpoints directly (no `gptqmodel`
runtime needed). `ov.convert_model` traces the whole thing; CPU and Arc iGPU
both compile it; logits match eager PyTorch within ~4e-4.

Happy to help shape a `Qwen3_5MoeOpenVINOConfig` + patcher PR mirroring the `qwen3_moe` one.

Reference: https://github.com/bob798/deepseek-v4-openvino (`src/qwen36/`)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the `optimum-cli export openvino` entry point and the existing `qwen3_moe` exporter configuration and patcher. Read the reference implementation under `src/qwen36/`, especially its state handling and model conversion path, and use `ov.convert_model` with CPU and Arc compilation as the validation path. Done means Qwen3.5 MoE checkpoints export successfully with traceable state IO and matching logits.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.