SciSharp / SciSharp/LLamaSharp

[BUG]: 0.24.0 CUDA detection bug

Open
#1,221 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
3.8k
Forks
508
Avg merge
1d 5h
Merged PRs (30d)
12

Description

Description

CPU will always load. When using fallback set to false and skipcheck() it will crash and throw the native api exception. Forcing CUDA does not work.

Temp Solution:
Use NextCoder model instead of my Qwen3 model and use version 0.23.0 of llamasharp with the cuda backend.
Llama loads up and shows backend_ptrs.size() = 2. This normally shows 1 when using 0.24.0.

Problem comes from 0.24.0 specifically.

Strange part:
I have 0.24.0 CUDA working on a MAUI application that is available on the Microsoft store so I know 0.24.0 works for CUDA but it uses a custom library that doesnt seem to work for the console/API server application.

Changing the targeted type from Any CPU to x64 is required to find the backend .dlls as well so x86 does not work correctly.

Reproduction Steps

Attempt to use Qwen3 model with latest llamasharp cuda backend.
Create a .NET 9 console application or API server and reference the latest CUDA backend.

Load the weights and look at where its assigned.

Environment & Configuration
  • Operating system: Windows 11
  • .NET runtime version: .NET 9
  • LLamaSharp version: 0.24.0
  • CUDA version (if you are using cuda backend): 12.6 & 12.9
  • CPU & GPU device: RTX 3090
Known Workarounds

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue in a .NET 9 console application or API server using LLamaSharp 0.24.0, the CUDA backend, and the Qwen3 model on Windows 11. Compare backend assignment and loading with version 0.23.0 and the working MAUI setup; done means CUDA is selected reliably without requiring CPU fallback or throwing the native API exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.