mudler / mudler/LocalAI

Quantization import always writes backend: llama-cpp, ignoring the backend that produced the model

Open Beginner friendly
#11,875 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
49.2k
Forks
4.5k
Avg merge
1d 3m
Merged PRs (30d)
239

Description

Summary

POST /api/quantization/jobs/:id/import always writes backend: llama-cpp into the generated model config, regardless of the quantization type the same service just produced. For any type stock llama.cpp cannot read, the imported model is not loadable.

Reproduction

Quantize through /app/quantize with a backend whose format stock llama.cpp does not know, then import the result. The generated YAML:

parameters:
    model: <name>/model-Q4_0_ROCMFP4_STRIX_LEAN.gguf
name: <name>
backend: llama-cpp          # <- always this
template:
    use_tokenizer_template: true
known_usecases:
    - chat
description: Fine-tuned model (GGUF)
options:
    - use_jinja:true

Loading it fails, because the tensor types are unknown to that backend. Correcting backend: by hand makes the same file work immediately.

Concretely this happened with the rocmfp4 backend (ROCmFP4 weight types, #11636), but the shape of the problem is general: the service knows which backend served StartQuantization, and discards that knowledge at import.

Suggestion

Carry the job's Backend into the imported config instead of defaulting. The value is already on the job record (schema.QuantizationJob.Backend), so the import has it in hand.

A second, smaller point in the same place: the description is hardcoded to Fine-tuned model (GGUF) for a model that was quantized, not fine-tuned.

Impact

Small in code, awkward in practice: the one path that produces a model and the one path that registers it disagree about how to run it, and the user finds out only when the model refuses to load.


Disclosure: this report was prepared with AI assistance (Claude); the import was run on a real host (LocalAI v4.9.0-305-g8351db5de, ROCm 7.x / gfx1151) and the YAML above is the file it wrote.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the POST /api/quantization/jobs/:id/import handler and inspect schema.QuantizationJob.Backend, then trace where the generated YAML sets backend and description. Reproduce an import with the rocmfp4 backend and verify the output preserves the job backend and no longer describes the model as fine-tuned.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.