intel / intel/AI-Playground

Chat template validation error when running Mistral‑7B‑Instruct‑v0.3‑Q4_K_S.gguf in llamaCPP backend

Open
#420 1 comment 0 reactions 0 assignees View on GitHub
bug stale
Dominant language
TypeScript
Stars
979
Forks
132
Avg merge
9h 23m
Merged PRs (30d)
8

Description

## Describe the bug

When loading Mistral‑7B‑Instruct‑v0.3‑Q4_K_S.gguf under the llamaCPP‑GGUF backend, Intel AI Playground v3.0.2 beta crashes immediately after sending the first user message.
The backend throws a chat template validation error, stating that conversation roles must alternate user/assistant/user/assistant/....

This happens because the GGUF file contains an incorrect LLaMA‑style chat template ([INST] ... [/INST]) that does not match the expected Mistral‑Instruct v0.3 format.
Since the Playground strictly enforces the embedded GGUF template and does not allow overriding it, the model cannot be used at all.

Other GGUF models (e.g., Llama‑3.1) work correctly, and the OpenVINO version of Mistral‑v0.3 loads and runs without issues.

## To Reproduce

Open Intel AI Playground v3.0.2 beta
Select backend: llamaCPP‑GGUF
Choose model: Mistral‑7B‑Instruct‑v0.3‑Q4_K_S.gguf (from the built‑in model list)
Type any message (e.g., “hi”)
The backend returns a 500 error and prints the chat template validation failure

## Expected behavior

The model should load and respond normally, or the Playground should allow overriding the incorrect GGUF chat template.
At minimum, the Playground should not crash due to invalid metadata inside a GGUF file.

## Screenshots

Here is the error displayed immediately after sending the first message:

The error shows repeated messages such as:
“Conversation roles must alternate user/assistant/user/assistant/...”

…and the full chat template is printed in the error log.

## Environment (please complete the following information):

OS: Windows 11
GPU: Intel Arc A750
CPU: Intel i3‑10100
RAM: 20 GB

Intel AI Playground Version: v3.0.2 beta

## Additional context

The llama.cpp backend successfully loads the model, offloads all 33 layers to the GPU, and initializes the KV cache without errors.

(From the log: “offloaded 33/33 layers to GPU”, “model loaded”, “server ready”)

The failure occurs only after the first chat request, not during model loading.
This confirms the issue is not related to GPU memory, Vulkan, or tensor loading.
The backend prints the embedded GGUF chat template during initialization:

Code
{{ bos_token }}{% for message in messages %} ... [INST] ... [/INST] ...

This template is incorrect for Mistral‑Instruct v0.3 and causes the strict role‑alternation validator to throw an exception.
The error repeats across multiple attempts:

Code
AI_RetryError: Failed after 3 attempts. Last error: Conversation roles must alternate...

This shows the Playground retries the request but cannot recover because the template is invalid
The error is thrown by the template engine before token generation begins, so the model never produces output.
Other GGUF models (e.g., Llama‑3.1‑8B‑Instruct) work correctly under the same backend, confirming the issue is specific to how the Playground handles this model’s embedded template.

OpenVINO models, including Mistral‑7B‑Instruct‑v0.3‑int4‑ov, load and run normally, which further isolates the problem to the GGUF template handling logic.

**Suggested fix**

Image

A user‑friendly solution would be to add a simple dropdown in Advanced Chat allowing users to choose the chat formatting style for GGUF models:

Auto (recommended)
Mistral format
Llama format
Qwen format
Custom (optional, hidden under an “Advanced” toggle)

This avoids exposing non‑technical users to raw template code, keeps the UI consistent with the rest of the Playground, and prevents crashes when a GGUF file contains an incorrect or incompatible chat template.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.