microsoft / microsoft/foundry-local
No integrity verification for cached models: a cache with wrong bytes reports healthy at every step and fails only at `LoadAsync`, with QNN 14001
- Dominant language
- C++
- Stars
- 2.6k
- Forks
- 369
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 39
Description
## Summary
A cached model can end up on disk at the correct size with the wrong bytes. Nothing in
the SDK notices. `IsCachedAsync`
returns `true`, `GetPathAsync` returns a populated directory, the reported size and
revision match, and a subsequent `DownloadAsync` returns immediately because the model
is considered present. The only symptom appears much later, at `LoadAsync`, as an
opaque QNN error 14001.
The result is that an application has no supported way to tell a corrupt cache apart
from an unsupported machine, and no supported way to repair one. Both present
identically: the model is cached, and it will not load.
## Environment
| | |
|---|---|
| SDK | `Microsoft.AI.Foundry.Local.WinML` 1.0.0 (Foundry Local Core 1.1.0) |
| Model | `qwen2.5-7b-instruct-qnn-npu:3` (alias `qwen2.5-7b`) |
| ORT | 1.23.x native bundled with the SDK, `Microsoft.ML.OnnxRuntime.Managed` 1.23.2 |
| Host | Snapdragon X Elite (X1P64100), Windows 11 arm64 10.0.26200 |
| EP | `MicrosoftCorporationII.WinML.Qualcomm.QNN.EP.1.8` 1.8.30.0, QnnHtp 2.40.0.0 |
| NPU driver | 30.0.220.3000, Compute DSP 2.0.4478.2200 |
| App | MSIX-packaged WinUI 3 / .NET 10 desktop app |
Two physically identical machines with byte-identical EP, driver, SoC, OS build and
RAM. The model loads on one and fails on the other.
## What we observed
On the failing machine, `LoadAsync` throws with ONNX Runtime text forwarded verbatim:
```
Failed to create context from binary
QnnBackendManager::LoadCachedQnnContextFromBuffer
Error code: 14001
```
Before reaching a cache theory we eliminated, on both machines:
- **Execution provider.** `EP.1.8` 1.8.30.0 is self-contained: it ships `QnnHtp.dll`,
`QnnHtpV73Stub.dll`, `libQnnHtpV73Skel.so` and `libqnnhtpv73.cat`. Nothing is missing
on the failing box.
- **Driver and SoC.** Identical, to the build number.
- **Signature catalogs.** No skel catalogs registered in `CatRoot` on either machine.
- **Model size and memory.** `qwen2.5-0.5b-instruct-qnn-npu`,
`qwen2.5-1.5b-instruct-qnn-npu` and `phi-3.5-mini-instruct-qnn-npu` all load
correctly **on the failing machine**, and all three are built with the same QNN SDK
2.37.1 as the 7B. So the EP, the driver, the SDK version and the NPU itself are all
working on that machine.
That left the bytes. We captured a manifest of the model cache from the working
machine (33 files, 6.81 GB, SHA-256 per file) and compared it against the failing
machine. Exactly one file differed:
```
CORRUPT context_1_ctx_qnn.bin, right size but different contents
470,421,504 bytes on both machines
```
Deleting the cached model and downloading it again fixed the machine.
We had watched Foundry's downloader abort and resume twice at roughly 59% earlier in
the same session, with 116 GB free on the target volume. That is the most likely
mechanism: a resumed range write that lands the file at full length with a bad
interior region. We want to be clear that this is inference and not something we
proved, which is why the report above is framed around detection rather than cause.
**These bytes were not written by our code.** The application never fetches model
files itself. Every download goes through `ModelVariant.DownloadAsync` with a progress
callback and nothing else; the only `HttpClient` in the codebase talks to an unrelated
local service. We also apply no timeout to the download, so we never cancel one on a
timer. The one way our process could interrupt a transfer is the user closing the app
mid-download, which is an ordinary real-world interruption rather than an induced one,
and is precisely the case a resume path needs to survive.
## Why this is hard to handle from an application
The failure surfaces roughly 470 MB and one process boundary away from its cause, and
every signal the SDK offers says the cache is fine:
1. **`IsCachedAsync` returns `true`.** It appears to check for presence, not integrity.
There is no checksum verification at the end of a download and no way to request one.
2. **`DownloadAsync` is a no-op once cached.** An app that catches the load failure and
retries the download gets an immediate success and the same broken bytes. There is
no `force` or `verify` parameter.
3. **There is no obvious supported repair.** `RemoveFromCacheAsync` exists but is not
discoverable as the documented remedy for this, and nothing in the load exception
suggests reaching for it.
4. **The error text names no file.** QNN 14001 identifies neither the context binary
nor the cache path, so the natural reading is "this machine cannot run this model",
which is exactly the wrong conclusion and sends you off measuring RAM and driver
versions, as it sent us.
To ship, we had to build all of this ourselves: capture a reference manifest on a known
good machine, ship it as an app asset, hash 6.81 GB of cache on the user's machine, and
add our own delete-and-redownload path. That is a lot of machinery to compensate for a
download that did not verify itself, and it only works because we happened to have a
second machine to take a reference from. An application shipping to real users has no
such machine.
## Steps to reproduce
The corruption itself depends on an interrupted transfer, so the deterministic
reproduction is to simulate the result rather than the cause. This demonstrates the
detection gap, which is the part we are asking about:
1. Download a QNN NPU model, for example `qwen2.5-7b-instruct-qnn-npu:3`, and confirm
it loads.
2. Locate the cache via `GetPathAsync`. In our case
`%USERPROFILE%\.\cache\models\Microsoft\qwen2.5-7b-instruct-qnn-npu-3\v3`.
3. Corrupt a context binary in place without changing its length:
```powershell
$f = "\context_1_ctx_qnn.bin"
$fs = [IO.File]::Open($f, 'Open', 'Write')
$fs.Seek(250MB, 'Begin') | Out-Null
$fs.Write((New-Object byte[] 4096), 0, 4096)
$fs.Dispose()
```
4. In a fresh process:
```csharp
var variant = await catalog.GetModelVariantAsync("qwen2.5-7b-instruct-qnn-npu:3");
Console.WriteLine(await variant.IsCachedAsync()); // true
await variant.DownloadAsync(); // returns immediately, no repair
Console.WriteLine(await variant.IsCachedAsync()); // still true
await variant.LoadAsync(); // throws, QNN 14001
```
**Expected:** either the download verifies and repairs the file, or `IsCachedAsync`
reports the cache as unusable, or the load error identifies the file that failed to
parse.
**Actual:** the cache reports healthy at every step and the load fails with an error
that points at the accelerator rather than at the bytes.
## Suggested fixes, roughly in order of value
1. **Verify after download.** Compare each file against the size and hash the service
already publishes in the model manifest, and fail the download rather than the load.
This alone would have turned a multi-day investigation into an error message.
2. **Make `DownloadAsync` repairable.** A `force: true` or `verify: true` option so an
application can recover without reverse-engineering the cache layout.
3. **Add an integrity API.** Something like `VerifyCacheAsync` returning the offending
files, so applications do not each have to reimplement hashing and ship their own
reference manifests.
4. **Name the file in the load failure.** Wrapping ORT's message with the context
binary path and the cache directory would let a developer reach the answer directly
from the exception.
5. **Harden the resume path.** If a resumed transfer cannot prove it continued from the
right offset, restart it instead of stitching.
Item 1 is the one that matters. The rest are mitigations for the case where it fails
anyway.
## Related
The same investigation ran into a second, separate problem worth mentioning because it
compounds this one: `FoundryLocalManager.IsInitialized` flips to `true` after
`CreateAsync` but **before** `DownloadAndRegisterEpsAsync` completes. Touching the
catalog in that window caches CPU-only variants for the lifetime of the process, so an
NPU model silently never gets offered. We work around it by awaiting EP registration
explicitly before any catalog access. Happy to file that separately if useful.
Contributor guide
Research direction
Start at ModelVariant.IsCachedAsync and DownloadAsync, then trace how the model manifest and cache files are handled before LoadAsync. Check whether the existing manifest exposes file hashes and whether cache or download tests cover interrupted transfers. Done means corrupted files are detected before load and the cache can be repaired or reported unusable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, csharp
- Domain
- ai-infra-agents, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100