huggingface / huggingface/candle
[feature] Weight quantization kernels: GPTQ/Marlin, AWQ, block-wise FP8
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
### Motivation
candle supports GGUF (`candle_core::quantized`), but the Hugging Face Hub is full
of `*-GPTQ` and `*-AWQ` 4-bit checkpoints that candle cannot load directly, plus
block-wise FP8 weights shipped by recent models (DeepSeek-V3, etc.). First-class
support would let users run these popular checkpoints without an offline
conversion step.
### Gap in candle
- No loader or dequant/GEMM kernel for GPTQ/Marlin or AWQ.
- No block-wise FP8 matmul (per-block scales).
### Proposed implementation
- 4-bit dequant + GEMM (Marlin-style) kernels in `candle-kernels`, surfaced via a
`QMatMul`-like type analogous to the existing GGUF path.
- Loaders reading `qweight/qzeros/scales/g_idx` (GPTQ) and
`qweight/scales/qzeros` (AWQ) from safetensors via `VarBuilder`.
- FP8: build on candle-core's `F8E4M3` dtype + a block-scaled GEMM.
### Use-case / example
```rust
// Load a 4-bit GPTQ checkpoint straight from the Hub, no pre-conversion.
let vb = VarBuilder::from_mmaped_safetensors(&["model.safetensors"], DType::F16, &device)?;
let q_proj = GptqLinear::load(vb.pp("model.layers.0.self_attn.q_proj"), /*bits=*/4, /*group=*/128)?;
let y = q_proj.forward(&x)?; // fused dequant + GEMM
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading candle_core::quantized and the existing GGUF path, then inspect candle-kernels for the relevant kernel entry points. Trace how VarBuilder loads safetensors and how the F8E4M3 dtype is exposed. Done means a supported path for GPTQ/Marlin, AWQ, and block-wise FP8 weights with fused or block-scaled matrix multiplication.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100