kvcache-ai / kvcache-ai/ktransformers

[Feature] Is there any chance to support unsloth's IQ1, IQ2 quantized DeepSeek 0528 and V3.1 models ?

Open
#1,513 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Checklist

- [x] 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [x] 2. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-English/Chinese content without translation may be closed.

### Motivation

I have tried running unsloth's DeepSeek-R1-0528-IQ2_XXS and DeepSeek-V3.1-IQ2_M using KT. Both models are not fully supported by KT yet where errors will be raised if one run them using KT. I only have a desktop computer with 256GB DDR5 memory. Thank you very much if you could reach me soon.

### Related resources

_No response_

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or entry points are named. Start by reproducing the reported errors with the unsloth DeepSeek-R1-0528-IQ2_XXS and DeepSeek-V3.1-IQ2_M models, then trace the unsupported paths. Done means both models run successfully in KT.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.