microsoft / microsoft/foundry-local
[Bug] Qwen2.5 QNN-NPU models ship a Phi-3 prompt template — control tokens absent from Qwen vocab, corrupting output and tool calls
- Dominant language
- C++
- Stars
- 2.6k
- Forks
- 369
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 39
Description
### Summary
Every `qwen2.5-*-qnn-npu` model in the catalog ships an `inference_model.json` whose `PromptTemplate` uses **Phi-3 control tokens** (`<|system|>`, `<|user|>`, `<|assistant|>`, `<|end|>`). Those tokens **do not exist in Qwen2.5's vocabulary**, so they are silently tokenized as ordinary BPE text instead of acting as turn delimiters.
The result is that every request to a Qwen2.5 NPU model is framed with a prompt the model was never trained on. Symptoms include intermittently corrupted special-token emission (e.g. `)((((` in place of ``) and degraded instruction-following.
The model directory already contains the **correct** template (`chat_template.jinja`, official Qwen ChatML with full tool support), but it is ignored in favour of the `PromptTemplate` written into `inference_model.json` at download time.
### Environment
| | |
|---|---|
| Foundry Local | `0.8.119` |
| OS | Windows 11, build `10.0.26200`, ARM64 |
| Device | Surface Pro 11 — Snapdragon X Elite `X1E80100` |
| NPU | Qualcomm Hexagon v73 (`QnnHtp.dll`, `QnnHtpV73Stub.dll` confirmed loaded) |
| Model | `qwen2.5-7b-instruct-qnn-npu:2` (also `1.5b`, `coder-0.5b`, `coder-1.5b`) |
### Evidence
**1. Shipped template uses Phi-3 tokens.**
`~/.foundry/cache/models/Microsoft/qwen2.5-7b-instruct-qnn-npu-2/v2/inference_model.json`:
```json
{
"Name": "qwen2.5-7b-instruct-qnn-npu:2",
"PromptTemplate": {
"system": "<|system|>You are Qwen, an AI assistant developed by Alibaba. {Content}<|end|>",
"user": "<|user|>{Content}<|end|>",
"assistant": "<|assistant|>{Content}<|end|>",
"prompt": "<|user|>{Content}<|end|><|assistant|>"
}
}
```
**2. Those tokens are not in the model's vocabulary.** Checked against the model's own `vocab.json` and `added_tokens.json`:
| Token used by template | in `vocab.json` | in `added_tokens.json` |
|---|---|---|
| `<\|system\|>` | ❌ | ❌ |
| `<\|user\|>` | ❌ | ❌ |
| `<\|assistant\|>` | ❌ | ❌ |
| `<\|end\|>` | ❌ | ❌ |
Qwen2.5's actual control tokens are present and unused by the template:
| Token | id |
|---|---|
| `<\|im_start\|>` | 151644 |
| `<\|im_end\|>` | 151645 |
| `` | 151657 |
| `` | 151658 |
**3. The correct template ships in the box and is ignored.** `chat_template.jinja` in the same directory is the official Qwen ChatML template, including full tool-calling support (injects `` signatures, renders `` blocks, handles the `tool` role via ``). The `PromptTemplate` schema in `inference_model.json` has only `system`/`user`/`assistant`/`prompt` and cannot express any of it — yet the catalog advertises these models as `Task: chat, tools`.
**4. CPU variants are packaged correctly.** `qwen3.5-0.8b-generic-cpu:2` gets it right, which shows the NPU packaging is a mistake rather than a deliberate choice:
```json
"user": "<|im_start|>user\n{Content}<|im_end|>"
```
### Reproduction
```bash
curl -s http://127.0.0.1:5273/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "qwen2.5-7b-instruct-qnn-npu:2",
"messages": [{"role":"user","content":"What is the weather in Barcelona? Use the tool."}],
"tools": [{"type":"function","function":{
"name":"get_weather",
"description":"Get current weather for a city",
"parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]
}'
```
**Actual** (as shipped) — control tokens intermittently corrupted:
```
)(((({"name": "get_weather", "arguments": {"city": "Barcelona"}}))))
```
Re-running the identical request sometimes yields well-formed `` tags instead, i.e. the defect is sampling-dependent, which is consistent with the model being driven off-distribution by unrecognised prompt framing.
### Fix / workaround
Replacing the `PromptTemplate` with correct ChatML resolves it:
```json
{
"Name": "qwen2.5-7b-instruct-qnn-npu:2",
"PromptTemplate": {
"system": "<|im_start|>system\n{Content}<|im_end|>",
"user": "<|im_start|>user\n{Content}<|im_end|>",
"assistant": "<|im_start|>assistant\n{Content}<|im_end|>",
"prompt": "<|im_start|>user\n{Content}<|im_end|>\n<|im_start|>assistant"
}
}
```
After patching and restarting the service, 5/5 identical requests produced clean, well-formed output with no corruption:
```
{"name": "get_weather", "arguments": {"city": "Barcelona"}}
```
Multi-turn conversation recall also behaves correctly, with no template-token leakage into responses.
Note this is a **catalog metadata** fix, not only a client one — `inference_model.json` is written at download time from catalog metadata (`sdk_v2/cpp/src/download/inference_model_writer.cc`), so a re-download reintroduces the bad template. Preferring the bundled `chat_template.jinja` when present would fix this class of bug generally, and is a prerequisite for real tool-calling support (cf. #874, #808).
### Related
- #874 — `tool_choice="auto"` returns tool calls as text with empty `tool_calls`. This issue is a plausible contributing root cause on NPU variants. Note that on `0.8.119` + QNN-NPU, `tool_choice="required"` does **not** yield structured `tool_calls` either, so the workaround documented in #874 does not apply here.
- #808 — Add ability to modify chat template kwargs.
Contributor guide
Research direction
Start in sdk_v2/cpp/src/download/inference_model_writer.cc and inspect how catalog metadata produces inference_model.json for the qwen2.5 QNN-NPU variants. Compare that output with the bundled chat_template.jinja and reproduce the curl tool-call request; done means a fresh download uses Qwen ChatML and repeated requests produce well-formed tool-call output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ai, backend-api-design
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100