Abnormally slow inference with gpt-oss-20b-int4-ov using OpenVINO backend
- Dominant language
- TypeScript
- Stars
- 979
- Forks
- 132
- Avg merge
- 9h 23m
- Merged PRs (30d)
- 8
Description
# Describe the bug
When using the OpenVINO backend in Intel AI Playground to run the model `gpt-oss-20b-int4-ov` (from https://huggingface.co/OpenVINO/gpt-oss-20b-int4-ov), the inference speed is abnormally slow (significantly slower than expected for an INT4 quantized 20B MoE model on Intel Arc / Core Ultra hardware).
Note: GPT-OSS 20B is officially supported in AI Playground via GGUF + Llama.cpp Vulkan, but the OpenVINO variant appears to be importable. However, its performance via OpenVINO is much worse than expected.
# Steps to reproduce
- Open Intel AI Playground (version 3.0.3-beta)
- Go to the Chat / LLM section
- Select OpenVINO as the backend/device
- Import or select the OpenVINO model `gpt-oss-20b-int4-ov`
- Start a conversation and generate responses
# Expected behavior
Token generation speed should be reasonable for an INT4 quantized 20B model on Intel hardware (e.g. 15–30+ tokens/sec on Arc GPU, depending on hardware).
OpenVINO should automatically use GPU acceleration (iGPU / dGPU) efficiently.
Actual behavior
- Token generation is extremely slow (e.g. 1–3 tokens/sec).
- The model is unable to apply chat template.
# Environment
AI Playground version: 3.0.3-beta
Operating System: Windows 11
Hardware:
CPU: 12th Gen Intel(R) Core(TM) i5-12600k
RAM: 64G
GPU: Intel Arc A770
Model: https://huggingface.co/OpenVINO/gpt-oss-20b-int4-ov
Backend: OpenVINO
# Additional context
- The same model in GGUF format with llama.cpp backend works much faster in AI Playground.
- No out-of-memory error. The model loads but performs poorly with OpenVINO.
Contributor guide
Assessment
This issue has not been assessed yet.