intel / intel/AI-Playground

Abnormally slow inference with gpt-oss-20b-int4-ov using OpenVINO backend

Open
#441 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
979
Forks
132
Avg merge
9h 23m
Merged PRs (30d)
8

Description

# Describe the bug

When using the OpenVINO backend in Intel AI Playground to run the model `gpt-oss-20b-int4-ov` (from https://huggingface.co/OpenVINO/gpt-oss-20b-int4-ov), the inference speed is abnormally slow (significantly slower than expected for an INT4 quantized 20B MoE model on Intel Arc / Core Ultra hardware).

Note: GPT-OSS 20B is officially supported in AI Playground via GGUF + Llama.cpp Vulkan, but the OpenVINO variant appears to be importable. However, its performance via OpenVINO is much worse than expected.

# Steps to reproduce

- Open Intel AI Playground (version 3.0.3-beta)
- Go to the Chat / LLM section
- Select OpenVINO as the backend/device
- Import or select the OpenVINO model `gpt-oss-20b-int4-ov`
- Start a conversation and generate responses

# Expected behavior

Token generation speed should be reasonable for an INT4 quantized 20B model on Intel hardware (e.g. 15–30+ tokens/sec on Arc GPU, depending on hardware).

OpenVINO should automatically use GPU acceleration (iGPU / dGPU) efficiently.

Actual behavior

- Token generation is extremely slow (e.g. 1–3 tokens/sec).
- The model is unable to apply chat template.

Image

# Environment

AI Playground version: 3.0.3-beta
Operating System: Windows 11
Hardware:
CPU: 12th Gen Intel(R) Core(TM) i5-12600k
RAM: 64G
GPU: Intel Arc A770

Model: https://huggingface.co/OpenVINO/gpt-oss-20b-int4-ov
Backend: OpenVINO

# Additional context

- The same model in GGUF format with llama.cpp backend works much faster in AI Playground.
- No out-of-memory error. The model loads but performs poorly with OpenVINO.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.