microsoft / microsoft/foundry-local
[Bug] qwen3.5-0.8b-cuda-gpu:3 fails to load: published vision.onnx ships ORT's own Memcpy nodes, so the memcpy pass collides on reload
- Dominant language
- C++
- Stars
- 2.6k
- Forks
- 369
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 39
Description
### Describe the issue
`qwen3.5-0.8b-cuda-gpu:3` cannot be loaded. It fails with:
```
genai_model_instance.cc:59 fl::GenAIModelInstance::GenAIModelInstance failed to load model
qwen3.5-0.8b-cuda-gpu:3: This is an invalid model.
Error: Duplicate definition of name (pad_CUDAExecutionProvider).
```
`qwen3.5-0.8b-generic-cpu:3` runs the same workload fine, and `:3` is already the latest
version of the CUDA variant, so there is nothing to upgrade to.
This looks like the same root cause as #1039 (`qwen3.5-9b-generic-gpu:3`, macOS, failing on
`image_grid_thw_CUDAExecutionProvider`), so please close this as a duplicate if you would
rather keep it in one place. I am filing separately because it is a different artifact on a
different platform and acceleration backend, and because I have an analysis of the cause
that I could not find recorded anywhere.
### The cause is in the published artifact, not in the caller
**`vision.onnx` inside the published download already contains ONNX Runtime's own
`Memcpy` nodes and their `_` outputs.** The graph appears to have been
exported *after* an ORT CUDA placement and memcpy-insertion pass, rather than before it.
Parsing the shipped `vision.onnx` of `qwen3.5-0.8b-cuda-gpu:3`:
```
Memcpy nodes baked into the graph: 55
Memcpy_token_218 MemcpyFromHost ['pad'] -> ['pad_CUDAExecutionProvider']
Memcpy_token_219 MemcpyFromHost ['prod'] -> ['prod_CUDAExecutionProvider']
Memcpy_token_257 MemcpyToHost ['val_1217_CUDAExecutionProvider'] -> ['val_1217']
...
```
So `pad_CUDAExecutionProvider` is not a duplicated name *in the file* — the file is a valid
graph, with no duplicate node names, node outputs or initializers. It is defined exactly
once, as the output of `Memcpy_token_218`. When ORT loads the model it runs its memcpy
transformer again over the same graph, regenerates the name `pad_CUDAExecutionProvider` for
the same tensor, and collides with the copy that was already baked in.
The contrast with a model that loads makes this concrete. `qwen3-vl-2b-instruct-cuda-gpu:2`
also ships baked `Memcpy` nodes (32 of them), but not one for `pad`:
| | `qwen3.5-0.8b-cuda-gpu:3` (fails) | `qwen3-vl-2b-instruct-cuda-gpu:2` (loads) |
| --- | --- | --- |
| baked `Memcpy` nodes | 55 | 32 |
| producer of `pad` | `node_pad` (Pad) | `node_pad` (Pad) |
| consumer of `pad` | `Memcpy_token_218` (MemcpyFromHost) | `n8` (Size) |
| `pad_CUDAExecutionProvider` | present, output of the baked memcpy | absent |
That also explains why #1039 sees a **`_CUDAExecutionProvider`** name while loading a
**`generic-gpu` (WebGPU)** variant: the CUDA-suffixed names are baked into the exported
artifact, so they travel with the model regardless of which EP is later used to run it.
To reproduce the analysis on any affected model (no GPU needed, no external data loaded):
```python
import onnx
g = onnx.load("vision.onnx", load_external_data=False).graph
print([f"{n.name} {n.op_type} {list(n.input)} -> {list(n.output)}"
for n in g.node if n.op_type.startswith("Memcpy")])
```
If this reading is right, the fix is in the model publishing pipeline — export the graph
before ORT's placement/memcpy pass — rather than in ORT or in Foundry Local. A defensive
alternative would be for the memcpy transformer to make generated names unique rather than
failing, but that would leave a redundant round trip in the graph.
### To reproduce
```python
# pip install foundry-local-sdk==2.0.1
from foundry_local_sdk import ChatSession, Configuration, FoundryLocalManager
manager = FoundryLocalManager(Configuration(app_name="repro"))
manager.download_and_register_eps()
model = manager.catalog.get_model("qwen3.5-0.8b") # resolves to qwen3.5-0.8b-cuda-gpu:3
print(model.id, model.info.runtime)
model.download()
model.load() # raises here
ChatSession(model)
```
Ruled out while investigating, in case it saves someone the time:
- **Not EP over-registration.** Registering only `CUDAExecutionProvider`
(`download_and_register_eps(names=["CUDAExecutionProvider"])`) fails identically.
Registering nothing fails earlier, with `requires CUDAExecutionProvider which is not
registered`, so the registration call is both necessary and not the trigger.
- **Not a corrupt local cache.** A clean download into a fresh `app_name` fails the same way.
- **Not an outdated variant.** `get_model_versions("qwen3.5-0.8b")` offers `-cuda-gpu:3` and
`:2`; `get_latest_version` returns `:3`, the one that fails.
- **Not model-family-wide.** `qwen3.5-0.8b-generic-cpu:3` and `qwen3-vl-2b-instruct-cuda-gpu:2`
both load and run on the same machine.
### Urgency
Not urgent. `-generic-cpu:3` is a working fallback on the same machine (3.5 s to load,
2.9 s to describe a 640x360 frame).
### Platform and architecture
Windows X64
### OS Version
Windows 11 Enterprise 26200
### Installation type
Released package/binary
### Foundry Local version
foundry-local-sdk 2.0.1 (Python, in-process); CLI 0.8.119
### API or surface area
Python SDK
### Acceleration backend
WebGPU/CUDA
Contributor guide
Research direction
Start with the Python SDK reproduction and inspect the published vision.onnx using the provided ONNX parsing snippet. Trace the model publishing pipeline that produces the CUDA artifact; done means a newly published qwen3.5-0.8b-cuda-gpu model loads without duplicate Memcpy-generated names while the CPU fallback remains unaffected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100