microsoft / microsoft/foundry-local
GPU (cuda) model variants produce deterministically corrupted output on H200/Hopper (ORT 1.26.0), CPU variants clean
- Dominant language
- C++
- Stars
- 2.6k
- Forks
- 369
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 39
Description
### What happened?
On an 8× **NVIDIA H200** (Hopper, `sm_90`) machine, every `*-cuda-gpu` model variant produces **deterministically corrupted output** (token doubling / gibberish), while the **CPU variants of the same models are perfectly clean**. The corruption is stable across runs at `temperature=0`, so it is not a sampling issue — it points to a CUDA compute/kernel problem in the bundled ONNX Runtime.
#### Clean vs. corrupted (identical prompt, `temperature=0`, `max_tokens=25`)
Prompt: `Count from 1 to 5`
| Variant | Output |
|---|---|
| `qwen2.5-coder-7b-instruct-generic-cpu` (CPU) | `Sure! Here is the count from 1 to 5:\n\n1\n2\n3\n4\n5` ✅ |
| `qwen2.5-1.5b-instruct-cuda-gpu` (GPU) | `Sure (lyyes))` ❌ |
The GPU result is **bit-for-bit identical across 3 consecutive runs** (`'Sure (lyyes))'` each time), confirming it is deterministic, not random sampling.
Other prompts show the classic **token-doubling** signature:
Prompt: `Count: 1 2 3 4 5` → GPU returns:
```
The:\n count is is 11 112 11131 111144 11
```
(note `is is`, `11 112 11131 111144` — tokens are being duplicated/interleaved)
Same corruption reproduced on `Phi-4-mini-instruct-cuda-gpu` as well (pure gibberish). It affects all GPU variants tested.
### Environment
- **Foundry Local**: `0.10.0+174be11ea7aeacd8d0d67b0ba1daebec615284b1`
- **Bundled ONNX Runtime**: `1.26.0` (from `/usr/local/lib/foundry-cli/libonnxruntime.so`), ONNX Runtime GenAI (Foundry build)
- **GPU**: 8× NVIDIA H200 (Hopper, compute capability 9.0)
- **NVIDIA driver**: `580.105.08`
- **CUDA EP libs present**: `libonnxruntime_providers_cuda.so`, `libonnxruntime-genai-cuda.so`
- **OS**: Linux
### How to reproduce
1. On an H200 (or other Hopper `sm_90`) host, `foundry model load qwen2.5-1.5b-instruct-cuda-gpu`.
2. Send a completion at `temperature=0`:
```bash
curl -s http://127.0.0.1:/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"qwen2.5-1.5b-instruct-cuda-gpu",
"messages":[{"role":"user","content":"Count from 1 to 5"}],
"max_tokens":25,"temperature":0}'
```
3. Observe garbled output; repeat and observe it is identical each time.
4. Load the CPU variant of any model and run the same prompt → clean output.
`nvidia-smi` confirms the model is resident in GPU memory and inference is fast (~2–3 s), so the CUDA EP is genuinely active — it is computing, just incorrectly.
### Analysis / suspected cause
The GenAI/CUDA execution path in the bundled **ONNX Runtime 1.26.0 (preview)** appears to miscompute on Hopper (`sm_90`). The corruption is:
- deterministic (independent of sampling),
- present on every GPU model variant,
- absent on all CPU variants,
- independent of cuDNN/cuBLAS/cudart minor versions (tested several CUDA 12.x runtime + cuDNN 9.x combinations on the linker path — corruption is unchanged).
This strongly suggests an ONNX Runtime / ONNX Runtime GenAI CUDA kernel issue on Hopper rather than a Foundry Local SDK/daemon-level bug.
### Questions for maintainers
1. Is this a known issue with the **ORT 1.26.0-preview** CUDA EP on Hopper (`sm_90`)?
2. **Would bumping the bundled ONNX Runtime / ONNX Runtime GenAI to a newer (or different stable) build fix this?** Since `flcore` links the bundled ORT `.so`s directly, users can't safely swap them without an ABI-matched build — a Foundry Local release with an updated ORT would be the clean fix.
3. Should this be tracked/forwarded to `microsoft/onnxruntime-genai` if it is confirmed to be a kernel-level bug?
Happy to run additional diagnostics on the H200 host.
Contributor guide
Research direction
Reproduce the failure with the listed foundry model load command and chat completion request on an H200, then inspect how flcore links /usr/local/lib/foundry-cli/libonnxruntime.so and the CUDA provider libraries. Compare the CUDA and CPU variants and use the deterministic corrupted output as the regression signal. Done means identifying a confirmed compatible ORT/GenAI fix or forwarding the issue with diagnostics to the appropriate runtime project.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ai, backend, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100