microsoft / microsoft/foundry-local
[Bug] qwen3.5-2b-text-cuda-gpu crashes on first message: CUDA failure 1 (invalid argument) in LinearAttention node
- Dominant language
- C++
- Stars
- 2.6k
- Forks
- 369
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 39
Description
### Describe the issue
Running `qwen3.5-2b-text-cuda-gpu:1` via `foundry run` and sending any chat message immediately throws an unhandled `OnnxRuntimeGenAIException` from a custom `LinearAttention` CUDA kernel. The model downloads and loads successfully, but generation fails on the very first prompt.
Error: error: IPC error 'op_handler_failed': Error from chat_completions command: Error: Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: Non-zero status code returned while running
LinearAttention node. Name:'/model/layers.0/linear_attn/LinearAttention' Status Message: CUDA failure 1: invalid argument ; GPU=0 ; hostname=LAPTOP ;
file=C:\a\_work\1\s\onnxruntime\contrib_ops\cuda\bert\linear_attention_impl.cu ; line=702 ; expr=attr_err;
at Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(IntPtr) + 0x47
at Microsoft.ML.OnnxRuntimeGenAI.Generator.AppendTokenSequences(Sequences) + 0x1f
at Microsoft.Neutron.OpenAI.Provider.OnnxChatGenerator..ctor(OnnxLoadedModel, GeneratorParams, ILogger, Sequences, NamedTensors) + 0x94
at Microsoft.Neutron.OpenAI.Provider.OnnxChatGenerator.CreateOnnxChatGenerator(ChatCompletionCreateRequestExtended, OnnxLoadedModel, AzureFoundryLocalModel, ITelemetry,
ILogger) + 0xa94
at Microsoft.AI.Foundry.Local.ChatClient.<>c__DisplayClass8_0.b__0(CancellationToken) + 0x2a
at Microsoft.Neutron.OpenAI.Provider.ChatCompletions.d__3.MoveNext() + 0x234
--- End of stack trace from previous location ---
at Microsoft.AI.Foundry.Local.ChatClient.d__8.MoveNext() + 0x2cb
--- End of stack trace from previous location ---
at Microsoft.AI.Foundry.Local.ChatClient.d__8.MoveNext() + 0x446
--- End of stack trace from previous location ---
at Microsoft.AI.Foundry.Local.NativeInterop.<>c__DisplayClass13_0.<b__2>d.MoveNext() + 0x467
--- End of stack trace from previous location ---
at Microsoft.AI.Foundry.Local.NativeInterop.<>c__DisplayClass13_0.<b__2>d.MoveNext() + 0x7d9
--- End of stack trace from previous location ---
at Microsoft.AI.Foundry.Local.NativeInterop.d__9.MoveNext() + 0xb8
### To reproduce
1. `foundry run qwen3.5-2b-text` (auto-selects `qwen3.5-2b-text-cuda-gpu:1`)
2. Wait for model download/load to complete
3. Type any message, e.g. `Hi`, and press enter
### Urgency
_No response_
### Platform and architecture
Windows X64
### OS Version
10.0.26200 N/A Build 26200
### Installation type
Released package/binary
### Foundry Local version
0.10.3
### API or surface area
CLI
### Hardware acceleration/backend
WebGPU/CUDA
### Backend/runtime version
CUDA release 12.8, V12.8.93
Contributor guide
Research direction
Start by reproducing the failure with `foundry run qwen3.5-2b-text` on Windows X64 using CUDA 12.8. Trace the chat_completions path through `OnnxChatGenerator` to the reported `LinearAttention` node and determine why the first prompt returns CUDA failure 1. Done means the model accepts an initial message without an unhandled exception on the reported backend.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ai, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100