microsoft / microsoft/foundry-local

GPU (cuda) model variants produce deterministically corrupted output on H200/Hopper (ORT 1.26.0), CPU variants clean

Open
#873 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.6k
Forks
369
Avg merge
2d 17h
Merged PRs (30d)
39

Description

### What happened?

On an 8× **NVIDIA H200** (Hopper, `sm_90`) machine, every `*-cuda-gpu` model variant produces **deterministically corrupted output** (token doubling / gibberish), while the **CPU variants of the same models are perfectly clean**. The corruption is stable across runs at `temperature=0`, so it is not a sampling issue — it points to a CUDA compute/kernel problem in the bundled ONNX Runtime.

#### Clean vs. corrupted (identical prompt, `temperature=0`, `max_tokens=25`)

Prompt: `Count from 1 to 5`

| Variant | Output |
|---|---|
| `qwen2.5-coder-7b-instruct-generic-cpu` (CPU) | `Sure! Here is the count from 1 to 5:\n\n1\n2\n3\n4\n5` ✅ |
| `qwen2.5-1.5b-instruct-cuda-gpu` (GPU) | `Sure (lyyes))` ❌ |

The GPU result is **bit-for-bit identical across 3 consecutive runs** (`'Sure (lyyes))'` each time), confirming it is deterministic, not random sampling.

Other prompts show the classic **token-doubling** signature:

Prompt: `Count: 1 2 3 4 5` → GPU returns:
```
The:\n count is is 11 112 11131 111144 11
```
(note `is is`, `11 112 11131 111144` — tokens are being duplicated/interleaved)

Same corruption reproduced on `Phi-4-mini-instruct-cuda-gpu` as well (pure gibberish). It affects all GPU variants tested.

### Environment

- **Foundry Local**: `0.10.0+174be11ea7aeacd8d0d67b0ba1daebec615284b1`
- **Bundled ONNX Runtime**: `1.26.0` (from `/usr/local/lib/foundry-cli/libonnxruntime.so`), ONNX Runtime GenAI (Foundry build)
- **GPU**: 8× NVIDIA H200 (Hopper, compute capability 9.0)
- **NVIDIA driver**: `580.105.08`
- **CUDA EP libs present**: `libonnxruntime_providers_cuda.so`, `libonnxruntime-genai-cuda.so`
- **OS**: Linux

### How to reproduce

1. On an H200 (or other Hopper `sm_90`) host, `foundry model load qwen2.5-1.5b-instruct-cuda-gpu`.
2. Send a completion at `temperature=0`:
```bash
curl -s http://127.0.0.1:/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"qwen2.5-1.5b-instruct-cuda-gpu",
"messages":[{"role":"user","content":"Count from 1 to 5"}],
"max_tokens":25,"temperature":0}'
```
3. Observe garbled output; repeat and observe it is identical each time.
4. Load the CPU variant of any model and run the same prompt → clean output.

`nvidia-smi` confirms the model is resident in GPU memory and inference is fast (~2–3 s), so the CUDA EP is genuinely active — it is computing, just incorrectly.

### Analysis / suspected cause

The GenAI/CUDA execution path in the bundled **ONNX Runtime 1.26.0 (preview)** appears to miscompute on Hopper (`sm_90`). The corruption is:
- deterministic (independent of sampling),
- present on every GPU model variant,
- absent on all CPU variants,
- independent of cuDNN/cuBLAS/cudart minor versions (tested several CUDA 12.x runtime + cuDNN 9.x combinations on the linker path — corruption is unchanged).

This strongly suggests an ONNX Runtime / ONNX Runtime GenAI CUDA kernel issue on Hopper rather than a Foundry Local SDK/daemon-level bug.

### Questions for maintainers

1. Is this a known issue with the **ORT 1.26.0-preview** CUDA EP on Hopper (`sm_90`)?
2. **Would bumping the bundled ONNX Runtime / ONNX Runtime GenAI to a newer (or different stable) build fix this?** Since `flcore` links the bundled ORT `.so`s directly, users can't safely swap them without an ABI-matched build — a Foundry Local release with an updated ORT would be the clean fix.
3. Should this be tracked/forwarded to `microsoft/onnxruntime-genai` if it is confirmed to be a kernel-level bug?

Happy to run additional diagnostics on the H200 host.

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the listed foundry model load command and chat completion request on an H200, then inspect how flcore links /usr/local/lib/foundry-cli/libonnxruntime.so and the CUDA provider libraries. Compare the CUDA and CPU variants and use the deterministic corrupted output as the regression signal. Done means identifying a confirmed compatible ORT/GenAI fix or forwarding the issue with diagnostics to the appropriate runtime project.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai, backend, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.