microsoft / microsoft/foundry-local

[Bug] Qwen2.5 QNN-NPU models ship a Phi-3 prompt template — control tokens absent from Qwen vocab, corrupting output and tool calls

Open
#1,001 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.6k
Forks
369
Avg merge
2d 17h
Merged PRs (30d)
39

Description

### Summary

Every `qwen2.5-*-qnn-npu` model in the catalog ships an `inference_model.json` whose `PromptTemplate` uses **Phi-3 control tokens** (`<|system|>`, `<|user|>`, `<|assistant|>`, `<|end|>`). Those tokens **do not exist in Qwen2.5's vocabulary**, so they are silently tokenized as ordinary BPE text instead of acting as turn delimiters.

The result is that every request to a Qwen2.5 NPU model is framed with a prompt the model was never trained on. Symptoms include intermittently corrupted special-token emission (e.g. `)((((` in place of ``) and degraded instruction-following.

The model directory already contains the **correct** template (`chat_template.jinja`, official Qwen ChatML with full tool support), but it is ignored in favour of the `PromptTemplate` written into `inference_model.json` at download time.

### Environment

| | |
|---|---|
| Foundry Local | `0.8.119` |
| OS | Windows 11, build `10.0.26200`, ARM64 |
| Device | Surface Pro 11 — Snapdragon X Elite `X1E80100` |
| NPU | Qualcomm Hexagon v73 (`QnnHtp.dll`, `QnnHtpV73Stub.dll` confirmed loaded) |
| Model | `qwen2.5-7b-instruct-qnn-npu:2` (also `1.5b`, `coder-0.5b`, `coder-1.5b`) |

### Evidence

**1. Shipped template uses Phi-3 tokens.**
`~/.foundry/cache/models/Microsoft/qwen2.5-7b-instruct-qnn-npu-2/v2/inference_model.json`:

```json
{
"Name": "qwen2.5-7b-instruct-qnn-npu:2",
"PromptTemplate": {
"system": "<|system|>You are Qwen, an AI assistant developed by Alibaba. {Content}<|end|>",
"user": "<|user|>{Content}<|end|>",
"assistant": "<|assistant|>{Content}<|end|>",
"prompt": "<|user|>{Content}<|end|><|assistant|>"
}
}
```

**2. Those tokens are not in the model's vocabulary.** Checked against the model's own `vocab.json` and `added_tokens.json`:

| Token used by template | in `vocab.json` | in `added_tokens.json` |
|---|---|---|
| `<\|system\|>` | ❌ | ❌ |
| `<\|user\|>` | ❌ | ❌ |
| `<\|assistant\|>` | ❌ | ❌ |
| `<\|end\|>` | ❌ | ❌ |

Qwen2.5's actual control tokens are present and unused by the template:

| Token | id |
|---|---|
| `<\|im_start\|>` | 151644 |
| `<\|im_end\|>` | 151645 |
| `` | 151657 |
| `` | 151658 |

**3. The correct template ships in the box and is ignored.** `chat_template.jinja` in the same directory is the official Qwen ChatML template, including full tool-calling support (injects `` signatures, renders `` blocks, handles the `tool` role via ``). The `PromptTemplate` schema in `inference_model.json` has only `system`/`user`/`assistant`/`prompt` and cannot express any of it — yet the catalog advertises these models as `Task: chat, tools`.

**4. CPU variants are packaged correctly.** `qwen3.5-0.8b-generic-cpu:2` gets it right, which shows the NPU packaging is a mistake rather than a deliberate choice:

```json
"user": "<|im_start|>user\n{Content}<|im_end|>"
```

### Reproduction

```bash
curl -s http://127.0.0.1:5273/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "qwen2.5-7b-instruct-qnn-npu:2",
"messages": [{"role":"user","content":"What is the weather in Barcelona? Use the tool."}],
"tools": [{"type":"function","function":{
"name":"get_weather",
"description":"Get current weather for a city",
"parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]
}'
```

**Actual** (as shipped) — control tokens intermittently corrupted:

```
)(((({"name": "get_weather", "arguments": {"city": "Barcelona"}}))))
```

Re-running the identical request sometimes yields well-formed `` tags instead, i.e. the defect is sampling-dependent, which is consistent with the model being driven off-distribution by unrecognised prompt framing.

### Fix / workaround

Replacing the `PromptTemplate` with correct ChatML resolves it:

```json
{
"Name": "qwen2.5-7b-instruct-qnn-npu:2",
"PromptTemplate": {
"system": "<|im_start|>system\n{Content}<|im_end|>",
"user": "<|im_start|>user\n{Content}<|im_end|>",
"assistant": "<|im_start|>assistant\n{Content}<|im_end|>",
"prompt": "<|im_start|>user\n{Content}<|im_end|>\n<|im_start|>assistant"
}
}
```

After patching and restarting the service, 5/5 identical requests produced clean, well-formed output with no corruption:

```

{"name": "get_weather", "arguments": {"city": "Barcelona"}}

```

Multi-turn conversation recall also behaves correctly, with no template-token leakage into responses.

Note this is a **catalog metadata** fix, not only a client one — `inference_model.json` is written at download time from catalog metadata (`sdk_v2/cpp/src/download/inference_model_writer.cc`), so a re-download reintroduces the bad template. Preferring the bundled `chat_template.jinja` when present would fix this class of bug generally, and is a prerequisite for real tool-calling support (cf. #874, #808).

### Related

- #874 — `tool_choice="auto"` returns tool calls as text with empty `tool_calls`. This issue is a plausible contributing root cause on NPU variants. Note that on `0.8.119` + QNN-NPU, `tool_choice="required"` does **not** yield structured `tool_calls` either, so the workaround documented in #874 does not apply here.
- #808 — Add ability to modify chat template kwargs.

Contributor guide

Open the contributing guide

Research direction

Start in sdk_v2/cpp/src/download/inference_model_writer.cc and inspect how catalog metadata produces inference_model.json for the qwen2.5 QNN-NPU variants. Compare that output with the bundled chat_template.jinja and reproduce the curl tool-call request; done means a fresh download uses Qwen ChatML and repeated requests produce well-formed tool-call output.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.