huggingface / huggingface/candle

[feature] Weight quantization kernels: GPTQ/Marlin, AWQ, block-wise FP8

Open
#3,650 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

### Motivation
candle supports GGUF (`candle_core::quantized`), but the Hugging Face Hub is full
of `*-GPTQ` and `*-AWQ` 4-bit checkpoints that candle cannot load directly, plus
block-wise FP8 weights shipped by recent models (DeepSeek-V3, etc.). First-class
support would let users run these popular checkpoints without an offline
conversion step.

### Gap in candle
- No loader or dequant/GEMM kernel for GPTQ/Marlin or AWQ.
- No block-wise FP8 matmul (per-block scales).

### Proposed implementation
- 4-bit dequant + GEMM (Marlin-style) kernels in `candle-kernels`, surfaced via a
`QMatMul`-like type analogous to the existing GGUF path.
- Loaders reading `qweight/qzeros/scales/g_idx` (GPTQ) and
`qweight/scales/qzeros` (AWQ) from safetensors via `VarBuilder`.
- FP8: build on candle-core's `F8E4M3` dtype + a block-scaled GEMM.

### Use-case / example
```rust
// Load a 4-bit GPTQ checkpoint straight from the Hub, no pre-conversion.
let vb = VarBuilder::from_mmaped_safetensors(&["model.safetensors"], DType::F16, &device)?;
let q_proj = GptqLinear::load(vb.pp("model.layers.0.self_attn.q_proj"), /*bits=*/4, /*group=*/128)?;
let y = q_proj.forward(&x)?; // fused dequant + GEMM
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading candle_core::quantized and the existing GGUF path, then inspect candle-kernels for the relevant kernel entry points. Trace how VarBuilder loads safetensors and how the F8E4M3 dtype is exposed. Done means a supported path for GPTQ/Marlin, AWQ, and block-wise FP8 weights with fused or block-scaled matrix multiplication.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.