huggingface / huggingface/optimum-intel
Native OpenVINO export for Qwen3.6 / Qwen3.5 MoE (model_type=qwen3_5_moe)
- Dominant language
- Jupyter Notebook
- Stars
- 620
- Forks
- 270
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 22
Description
Qwen released the Qwen3.6-35B-A3B family (registered as `model_type="qwen3_5_moe"`,
class `Qwen3_5MoeForConditionalGeneration`). As of `optimum-intel` 1.27.0,
`optimum-cli export openvino` fails before it even reaches the exporter:
```
ValueError: The checkpoint you are trying to load has model type `qwen3_5_moe`
but Transformers does not recognize this architecture.
```
with `transformers>=5.5` (which does register the architecture) the export then
fails at the OpenVINO exporter config map, which registers `qwen2`, `qwen2_moe`,
`qwen3`, `qwen3_moe`, `qwen2_vl`, `qwen2_5_vl` but not `qwen3_5_moe`.
### What makes this architecture non-trivial
Unlike `qwen3_moe`, this family is a **3:1 hybrid**: 3 Gated-DeltaNet
(linear-attention) layers per 1 full GQA-attention layer. The DeltaNet layers:
- maintain recurrent state (`conv_state` + `recurrent_state`) updated in place
inside a `Cache` object — not graph-traceable as written;
- default to CUDA-only kernels (`flash-linear-attention`, `causal-conv1d`) with
pure-torch fallbacks (`torch_recurrent_gated_delta_rule`,
`torch_causal_conv1d_update`) that *are* traceable.
A working OpenVINO export needs: (1) a model-patcher that forces the torch
fallbacks, (2) recurrent/conv state lifted to explicit model inputs/outputs
(stateful or KV-style), (3) the MoE routing expressed graph-friendly.
### Reference implementation
I have a self-contained, OV-traceable port: `QwenGatedDeltaNet`
with functional state IO, GQA attention with explicit KV cache, MoE as
compute-all+mask (single IR) **or** a backbone + per-expert split, plus a GPTQ
int4 dequantizer that reads the community checkpoints directly (no `gptqmodel`
runtime needed). `ov.convert_model` traces the whole thing; CPU and Arc iGPU
both compile it; logits match eager PyTorch within ~4e-4.
Happy to help shape a `Qwen3_5MoeOpenVINOConfig` + patcher PR mirroring the `qwen3_moe` one.
Reference: https://github.com/bob798/deepseek-v4-openvino (`src/qwen36/`)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the `optimum-cli export openvino` entry point and the existing `qwen3_moe` exporter configuration and patcher. Read the reference implementation under `src/qwen36/`, especially its state handling and model conversion path, and use `ov.convert_model` with CPU and Arc compilation as the validation path. Done means Qwen3.5 MoE checkpoints export successfully with traceable state IO and matching logits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100