microsoft / microsoft/foundry-local

[Bug] qwen3.5-0.8b-cuda-gpu:3 fails to load: published vision.onnx ships ORT's own Memcpy nodes, so the memcpy pass collides on reload

Open
#1,075 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.6k
Forks
369
Avg merge
2d 17h
Merged PRs (30d)
39

Description

### Describe the issue

`qwen3.5-0.8b-cuda-gpu:3` cannot be loaded. It fails with:

```
genai_model_instance.cc:59 fl::GenAIModelInstance::GenAIModelInstance failed to load model
qwen3.5-0.8b-cuda-gpu:3: This is an invalid model.
Error: Duplicate definition of name (pad_CUDAExecutionProvider).
```

`qwen3.5-0.8b-generic-cpu:3` runs the same workload fine, and `:3` is already the latest
version of the CUDA variant, so there is nothing to upgrade to.

This looks like the same root cause as #1039 (`qwen3.5-9b-generic-gpu:3`, macOS, failing on
`image_grid_thw_CUDAExecutionProvider`), so please close this as a duplicate if you would
rather keep it in one place. I am filing separately because it is a different artifact on a
different platform and acceleration backend, and because I have an analysis of the cause
that I could not find recorded anywhere.

### The cause is in the published artifact, not in the caller

**`vision.onnx` inside the published download already contains ONNX Runtime's own
`Memcpy` nodes and their `_` outputs.** The graph appears to have been
exported *after* an ORT CUDA placement and memcpy-insertion pass, rather than before it.

Parsing the shipped `vision.onnx` of `qwen3.5-0.8b-cuda-gpu:3`:

```
Memcpy nodes baked into the graph: 55
Memcpy_token_218 MemcpyFromHost ['pad'] -> ['pad_CUDAExecutionProvider']
Memcpy_token_219 MemcpyFromHost ['prod'] -> ['prod_CUDAExecutionProvider']
Memcpy_token_257 MemcpyToHost ['val_1217_CUDAExecutionProvider'] -> ['val_1217']
...
```

So `pad_CUDAExecutionProvider` is not a duplicated name *in the file* — the file is a valid
graph, with no duplicate node names, node outputs or initializers. It is defined exactly
once, as the output of `Memcpy_token_218`. When ORT loads the model it runs its memcpy
transformer again over the same graph, regenerates the name `pad_CUDAExecutionProvider` for
the same tensor, and collides with the copy that was already baked in.

The contrast with a model that loads makes this concrete. `qwen3-vl-2b-instruct-cuda-gpu:2`
also ships baked `Memcpy` nodes (32 of them), but not one for `pad`:

| | `qwen3.5-0.8b-cuda-gpu:3` (fails) | `qwen3-vl-2b-instruct-cuda-gpu:2` (loads) |
| --- | --- | --- |
| baked `Memcpy` nodes | 55 | 32 |
| producer of `pad` | `node_pad` (Pad) | `node_pad` (Pad) |
| consumer of `pad` | `Memcpy_token_218` (MemcpyFromHost) | `n8` (Size) |
| `pad_CUDAExecutionProvider` | present, output of the baked memcpy | absent |

That also explains why #1039 sees a **`_CUDAExecutionProvider`** name while loading a
**`generic-gpu` (WebGPU)** variant: the CUDA-suffixed names are baked into the exported
artifact, so they travel with the model regardless of which EP is later used to run it.

To reproduce the analysis on any affected model (no GPU needed, no external data loaded):

```python
import onnx
g = onnx.load("vision.onnx", load_external_data=False).graph
print([f"{n.name} {n.op_type} {list(n.input)} -> {list(n.output)}"
for n in g.node if n.op_type.startswith("Memcpy")])
```

If this reading is right, the fix is in the model publishing pipeline — export the graph
before ORT's placement/memcpy pass — rather than in ORT or in Foundry Local. A defensive
alternative would be for the memcpy transformer to make generated names unique rather than
failing, but that would leave a redundant round trip in the graph.

### To reproduce

```python
# pip install foundry-local-sdk==2.0.1
from foundry_local_sdk import ChatSession, Configuration, FoundryLocalManager

manager = FoundryLocalManager(Configuration(app_name="repro"))
manager.download_and_register_eps()

model = manager.catalog.get_model("qwen3.5-0.8b") # resolves to qwen3.5-0.8b-cuda-gpu:3
print(model.id, model.info.runtime)
model.download()
model.load() # raises here
ChatSession(model)
```

Ruled out while investigating, in case it saves someone the time:

- **Not EP over-registration.** Registering only `CUDAExecutionProvider`
(`download_and_register_eps(names=["CUDAExecutionProvider"])`) fails identically.
Registering nothing fails earlier, with `requires CUDAExecutionProvider which is not
registered`, so the registration call is both necessary and not the trigger.
- **Not a corrupt local cache.** A clean download into a fresh `app_name` fails the same way.
- **Not an outdated variant.** `get_model_versions("qwen3.5-0.8b")` offers `-cuda-gpu:3` and
`:2`; `get_latest_version` returns `:3`, the one that fails.
- **Not model-family-wide.** `qwen3.5-0.8b-generic-cpu:3` and `qwen3-vl-2b-instruct-cuda-gpu:2`
both load and run on the same machine.

### Urgency

Not urgent. `-generic-cpu:3` is a working fallback on the same machine (3.5 s to load,
2.9 s to describe a 640x360 frame).

### Platform and architecture

Windows X64

### OS Version

Windows 11 Enterprise 26200

### Installation type

Released package/binary

### Foundry Local version

foundry-local-sdk 2.0.1 (Python, in-process); CLI 0.8.119

### API or surface area

Python SDK

### Acceleration backend

WebGPU/CUDA

Contributor guide

Open the contributing guide

Research direction

Start with the Python SDK reproduction and inspect the published vision.onnx using the provided ONNX parsing snippet. Trace the model publishing pipeline that produces the CUDA artifact; done means a newly published qwen3.5-0.8b-cuda-gpu model loads without duplicate Memcpy-generated names while the CPU fallback remains unaffected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.