Quantization import always writes backend: llama-cpp, ignoring the backend that produced the model
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 49.2k
- Forks
- 4.5k
- Avg merge
- 1d 3m
- Merged PRs (30d)
- 239
Description
Summary
POST /api/quantization/jobs/:id/import always writes backend: llama-cpp into the generated model config, regardless of the quantization type the same service just produced. For any type stock llama.cpp cannot read, the imported model is not loadable.
Reproduction
Quantize through /app/quantize with a backend whose format stock llama.cpp does not know, then import the result. The generated YAML:
parameters:
model: <name>/model-Q4_0_ROCMFP4_STRIX_LEAN.gguf
name: <name>
backend: llama-cpp # <- always this
template:
use_tokenizer_template: true
known_usecases:
- chat
description: Fine-tuned model (GGUF)
options:
- use_jinja:true
Loading it fails, because the tensor types are unknown to that backend. Correcting backend: by hand makes the same file work immediately.
Concretely this happened with the rocmfp4 backend (ROCmFP4 weight types, #11636), but the shape of the problem is general: the service knows which backend served StartQuantization, and discards that knowledge at import.
Suggestion
Carry the job's Backend into the imported config instead of defaulting. The value is already on the job record (schema.QuantizationJob.Backend), so the import has it in hand.
A second, smaller point in the same place: the description is hardcoded to Fine-tuned model (GGUF) for a model that was quantized, not fine-tuned.
Impact
Small in code, awkward in practice: the one path that produces a model and the one path that registers it disagree about how to run it, and the user finds out only when the model refuses to load.
Disclosure: this report was prepared with AI assistance (Claude); the import was run on a real host (LocalAI v4.9.0-305-g8351db5de, ROCm 7.x / gfx1151) and the YAML above is the file it wrote.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the POST /api/quantization/jobs/:id/import handler and inspect schema.QuantizationJob.Backend, then trace where the generated YAML sets backend and description. Reproduce an import with the rocmfp4 backend and verify the output preserves the job backend and no longer describes the model as fine-tuned.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100