microsoft / microsoft/foundry-local

[Bug] Foundry Local: QNN NPU EP shows heavy CPU usage during multi-round tool-calling, GPU EP does not

Open
#1,081 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
2.6k
Forks
369
Avg merge
2d 17h
Merged PRs (30d)
39

Description

### Describe the issue

During multi-round tool-calling (agentic loop: call model → get tool_calls → feed tool results back → call model again, repeated up to N rounds), the QNN NPU execution provider shows sustained high CPU utilization (~74-84% across all logical cores) alongside only partial NPU compute utilization (~43-50%, sawtooth). The same model, same prompts, same multi-round loop code path run against the GPU (DirectML) execution provider instead shows the expected "healthy offload" pattern: GPU saturated near 99%, CPU at only ~16-20%.

This suggests a QNN NPU EP-specific issue in how repeated in-session calls are handled (e.g. per-round re-quantization, dispatch overhead, or CPU-side fallback ops that compound across rounds) — not a problem with the model weights or the general orchestration code, since GPU with the literal same code shows no spike, and a single non-looping NPU call also shows no spike.

on the :2 version of qwen2.5-7b this behavior did not happen

## Environment

- Foundry Local: 0.10.3 (also reproduced pre/post the 2026-09
`foundry-local-sdk`/`Microsoft.AI.Foundry.Local.WinML` upgrade — this is
not new/introduced by that upgrade, present both before and after)
- Model: `qwen2.5-7b-instruct-qnn-npu:3` vs `qwen2.5-7b-instruct-generic-gpu`
(DirectML) / `qwen2.5-7b-instruct-openvino-gpu`
- SDKs affected: both `foundry-local-sdk` (Node.js) and
`Microsoft.AI.Foundry.Local.WinML` (.NET) — not SDK-specific
- Devices reproduced on:
- Snapdragon X2 Elite (Surface Laptop for Business 13.8in 8th Ed),
Windows 11 ARM64, NPU driver 30.0.225.10000 (5/21/2026)
- A second, untouched Snapdragon X (original) Surface device, using the
real shipped Zava Dental production app (native .NET/WinUI, same
`Microsoft.AI.Foundry.Local.WinML` SDK family)
- Intel Core Ultra 5 device: GPU path shows the healthy pattern
(CPU ~8-12%, GPU ~95%) for comparison — Intel does not have a QNN NPU
to directly compare, included here as supporting cross-platform GPU
evidence that the "high CPU during multi-round loop" pattern is not a
generic property of the multi-round tool-calling loop itself.

### To reproduce

Load a model's QNN NPU variant (e.g. `qwen2.5-7b-instruct-qnn-npu:3`)
via the SDK (`FoundryLocalManager` / catalog / `model.load()` /
`createChatClient()`).
2. Run a multi-round tool-calling loop: send a prompt with 3-4 tool
schemas offered, receive a tool call, feed a canned tool result back as
a new message, call the model again — repeat for several rounds
(reproduced with up to 5 rounds).
3. Watch Task Manager's CPU and NPU graphs during the run.
4. **Observed:** CPU pinned at ~74-84% across all logical cores; NPU
compute utilization only ~43-50%, sawtoothing.
5. Repeat steps 1-4 but load the GPU (DirectML) variant instead of the
NPU variant, same loop, same prompts.
6. **Observed:** GPU compute saturates near 99%; CPU drops to ~16-20%.
7. Repeat steps 1-2 but with a **single**, non-looping `completeChat()`
call (no round loop) on the NPU variant, using the same system prompt
and same 4 tool schemas.
8. **Observed:** CPU only reaches ~15-37% here (proportional to the NPU's
prefill/generation bumps) — the healthy pattern, same shape as GPU.

## Expected behavior

CPU utilization during NPU-backed multi-round tool-calling should look
similar to the GPU-backed case: NPU compute saturated, CPU largely idle
(aside from normal host-side orchestration overhead). The entire purpose
of an NPU execution provider is to offload compute away from the CPU.

## Actual behavior

CPU utilization is high (~74-84%, all cores) specifically when: (a) the
QNN NPU EP is used, **and** (b) the call is part of a multi-round loop
(≥2 sequential `completeChat()` calls in the same session/context, as
happens naturally in tool-calling). Neither condition alone reproduces
the issue:
- NPU + single-shot call → healthy CPU (~15-37%)
- GPU + multi-round loop → healthy CPU (~16-20%)
- NPU + multi-round loop → **high CPU (~74-84%)**

### Urgency

This negates any of the NPU value since the CPU is pegged at ~90% during the LLM run along with the NPU from 40-90%

### Platform and architecture

Windows ARM64

### OS Version

WIndows 11 26H1, Windows 11 preview, Windows 11 25H@

### Installation type

Released package/binary

### Foundry Local version

CLI 0.10.3 and latest SDK

### API or surface area

JavaScript SDK

### Hardware acceleration/backend

WinML EPs

### Backend/runtime version

This applies to all backends and languages (.net and JS) and CLI

Contributor guide

Open the contributing guide

Research direction

No source file or failing test is named. Start by reproducing the multi-round loop through FoundryLocalManager, model.load(), createChatClient(), and completeChat(), then compare QNN NPU with DirectML GPU and a single NPU call. Done means the QNN-specific CPU behavior is isolated and a regression test or captured comparison demonstrates the expected result.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
backend, machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.