microsoft / microsoft/foundry-local

[Bug] qwen3.5-2b-text-cuda-gpu crashes on first message: CUDA failure 1 (invalid argument) in LinearAttention node

Open
#1,031 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
2.6k
Forks
369
Avg merge
2d 17h
Merged PRs (30d)
39

Description

### Describe the issue

Running `qwen3.5-2b-text-cuda-gpu:1` via `foundry run` and sending any chat message immediately throws an unhandled `OnnxRuntimeGenAIException` from a custom `LinearAttention` CUDA kernel. The model downloads and loads successfully, but generation fails on the very first prompt.

Error: error: IPC error 'op_handler_failed': Error from chat_completions command: Error: Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: Non-zero status code returned while running
LinearAttention node. Name:'/model/layers.0/linear_attn/LinearAttention' Status Message: CUDA failure 1: invalid argument ; GPU=0 ; hostname=LAPTOP ;
file=C:\a\_work\1\s\onnxruntime\contrib_ops\cuda\bert\linear_attention_impl.cu ; line=702 ; expr=attr_err;
at Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(IntPtr) + 0x47
at Microsoft.ML.OnnxRuntimeGenAI.Generator.AppendTokenSequences(Sequences) + 0x1f
at Microsoft.Neutron.OpenAI.Provider.OnnxChatGenerator..ctor(OnnxLoadedModel, GeneratorParams, ILogger, Sequences, NamedTensors) + 0x94
at Microsoft.Neutron.OpenAI.Provider.OnnxChatGenerator.CreateOnnxChatGenerator(ChatCompletionCreateRequestExtended, OnnxLoadedModel, AzureFoundryLocalModel, ITelemetry,
ILogger) + 0xa94
at Microsoft.AI.Foundry.Local.ChatClient.<>c__DisplayClass8_0.b__0(CancellationToken) + 0x2a
at Microsoft.Neutron.OpenAI.Provider.ChatCompletions.d__3.MoveNext() + 0x234
--- End of stack trace from previous location ---
at Microsoft.AI.Foundry.Local.ChatClient.d__8.MoveNext() + 0x2cb
--- End of stack trace from previous location ---
at Microsoft.AI.Foundry.Local.ChatClient.d__8.MoveNext() + 0x446
--- End of stack trace from previous location ---
at Microsoft.AI.Foundry.Local.NativeInterop.<>c__DisplayClass13_0.<b__2>d.MoveNext() + 0x467
--- End of stack trace from previous location ---
at Microsoft.AI.Foundry.Local.NativeInterop.<>c__DisplayClass13_0.<b__2>d.MoveNext() + 0x7d9
--- End of stack trace from previous location ---
at Microsoft.AI.Foundry.Local.NativeInterop.d__9.MoveNext() + 0xb8

### To reproduce

1. `foundry run qwen3.5-2b-text` (auto-selects `qwen3.5-2b-text-cuda-gpu:1`)
2. Wait for model download/load to complete
3. Type any message, e.g. `Hi`, and press enter

### Urgency

_No response_

### Platform and architecture

Windows X64

### OS Version

10.0.26200 N/A Build 26200

### Installation type

Released package/binary

### Foundry Local version

0.10.3

### API or surface area

CLI

### Hardware acceleration/backend

WebGPU/CUDA

### Backend/runtime version

CUDA release 12.8, V12.8.93

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with `foundry run qwen3.5-2b-text` on Windows X64 using CUDA 12.8. Trace the chat_completions path through `OnnxChatGenerator` to the reported `LinearAttention` node and determine why the first prompt returns CUDA failure 1. Done means the model accepts an initial message without an unhandled exception on the reported backend.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.