mudler / mudler/LocalAI

localai cannot run the recommended model

Open
#12,081 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug unconfirmed
Dominant language
Go
Stars
49.2k
Forks
4.5k
Avg merge
1d 3m
Merged PRs (30d)
239

Description

LocalAI version:

quay.io/go-skynet/local-ai:master-gpu-nvidia-cuda-13 pulled today

Environment, CPU architecture, OS, and Version:

Linux beast.local 7.2.5-200.fc44.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 11 15:11:05 UTC 2026 x86_64 GNU/Linux
Tue Sep 15 21:41:49 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 615.71.09              KMD Version: 615.71.09     CUDA UMD Version: 13.4     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4070 Ti     Off |   00000000:05:00.0 Off |                  N/A |
|  0%   31C    P8              6W /  285W |     164MiB /  12282MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            1846      G   /usr/bin/gnome-shell                    121MiB |
|    0   N/A  N/A            3407      G   /usr/bin/Xwayland                         3MiB |
+-----------------------------------------------------------------------------------------+

Describe the bug

On a fresh install, localai recommends spark-x2.5-4b. I download and run it, and I get

Error: failed to load model with internal loader: could not load model: rpc error: code = Internal desc = Failed to load model: /models/llama-cpp/models/Spark-X2.5-4B-Q4_K_M/Spark-X2.5-4B-Q4_K_M.gguf. Error: llama_model_load: error loading model: unknown model architecture: 'spark2_5'; llama_model_load_from_file_impl: failed to load model; llama_model_load: error loading model: unknown model architecture: 'spark2_5'; llama_model_load_from_file_impl: failed to load model

To Reproduce

Fresh install, install and start the recommended model. Fails consistently

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the fresh-install failure through the internal loader and inspect the /models/llama-cpp/models/Spark-X2.5-4B-Q4_K_M path and the reported spark2_5 architecture error. Compare the recommended spark-x2.5-4b model with the architectures supported by the loader; done means the recommended model starts without a loading error.

Written by the indexing model from the issue text.

Assessment

Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.