kvcache-ai / kvcache-ai/ktransformers
[Feature] Is there any chance to support unsloth's IQ1, IQ2 quantized DeepSeek 0528 and V3.1 models ?
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Checklist
- [x] 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [x] 2. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-English/Chinese content without translation may be closed.
### Motivation
I have tried running unsloth's DeepSeek-R1-0528-IQ2_XXS and DeepSeek-V3.1-IQ2_M using KT. Both models are not fully supported by KT yet where errors will be raised if one run them using KT. I only have a desktop computer with 256GB DDR5 memory. Thank you very much if you could reach me soon.
### Related resources
_No response_
Contributor guide
Research direction
No source files, tests, or entry points are named. Start by reproducing the reported errors with the unsloth DeepSeek-R1-0528-IQ2_XXS and DeepSeek-V3.1-IQ2_M models, then trace the unsupported paths. Done means both models run successfully in KT.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100