SciSharp / SciSharp/LLamaSharp
Examples don't run with CUDA12
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 3.8k
- Forks
- 508
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 12
Description
I have CUDA 12, I know it works because I ran custom pytorch model for other projects.
But when I try to run LlamaSharp.Examples with LLamaSharp.Backend.Cpu, it works fine. But when I try to use it with LLamaSharp.Backend.Cuda12, it crash right away with the following error:
System.TypeInitializationException
HResult=0x80131534
Message=The type initializer for 'LLama.Native.NativeApi' threw an exception.
Source=LLamaSharp
StackTrace:
at LLama.Native.NativeApi.llama_empty_call() in C:\work\Projects\LLamaSharp\LLama\Native\NativeApi.cs:line 27
at Program.<<Main>$>d__0.MoveNext() in C:\work\Projects\LLamaSharp\LLama.Examples\Program.cs:line 24
This exception was originally thrown at this call stack:
LLama.Native.NativeApi.NativeApi() in NativeApi.Load.cs
Inner Exception 1:
RuntimeError: The native library cannot be correctly loaded. It could be one of the following reasons:
1. No LLamaSharp backend was installed. Please search LLamaSharp.Backend and install one of them.
2. You are using a device with only CPU but installed cuda backend. Please install cpu backend instead.
3. One of the dependency of the native library is missed. Please use `ldd` on linux, `dumpbin` on windows and `otool`to check if all the dependency of the native library is satisfied. Generally you could find the libraries under your output folder.
4. Try to compile llama.cpp yourself to generate a libllama library, then use `LLama.Native.NativeLibraryConfig.WithLibrary` to specify it at the very beginning of your code. For more informations about compilation, please refer to LLamaSharp repo on github.
I tried running the project in Debug, also with GPU in the configuration manager, tried running it in .net 8, .net 6, and all combinations but always the same error. I am running the latest version for the nuget packages, 0.10.0.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with LLama/Native/NativeApi.cs at line 27 and NativeApi.Load.cs, then reproduce from LLama.Examples/Program.cs at line 24 using LLamaSharp.Backend.Cuda12. Compare the CUDA12 native library dependencies with the working CPU backend. Done means the CUDA12 example starts without the native-library initialization exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100