[feature request] Mixed quantizations.
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 4.7k
- Forks
- 537
- Avg merge
- 12h 12m
- Merged PRs (30d)
- 4
Description
From my own experience in text generation models, I found out that quantizing the output and embed tensors to f16 and the other tensors to q6_k (or q5_k) gives smaller files and better results that quantizing everything to q8_0.
Usually in my tests, i quantize the output and embed tensors to F16 and the inner ones to q5_k q6_k and q8_0.
The results I then test using llama.cpp qhich is quite fast even on cpu only...
Can you please add the feature to ctensor2 too?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The request targets mixed quantization in ctensor2 but names no files, tests, or implementation entry points. Start by locating ctensor2's quantization path and compare its behavior with the llama.cpp results described; done means supporting F16 output and embedding tensors alongside q5_k, q6_k, or q8_0 inner tensors with validated model results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100