huggingface / huggingface/candle

`candle-transformers`: dispatch GGUF checkpoints by `general.architecture`

Open
#3,783 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

### Problem

`candle-transformers` ships a growing set of `quantized_*` modules —
`quantized_llama`, `quantized_qwen2`, `quantized_qwen3`, `quantized_qwen3_moe`,
`quantized_gemma3`, `quantized_phi3`, `quantized_glm4`, … — but no way to load
"whatever GGUF this is". Every consumer writes the same dispatcher.

Two things make that dispatcher more annoying than it should be:

**Signatures diverge.** They are close enough to look uniform and different
enough to need a wrapper enum:

```rust
quantized_llama::ModelWeights::from_gguf(ct, reader, device)
quantized_qwen2::ModelWeights::from_gguf(ct, reader, device)
quantized_qwen3::ModelWeights::from_gguf(ct, reader, device)
quantized_gemma3::ModelWeights::from_gguf(ct, reader, device)
quantized_qwen3_moe::GGUFQWenMoE::from_gguf(ct, reader, device, dtype) // + dtype
quantized_phi3::ModelWeights::from_gguf(use_flash_attn, ct, reader, device) // leading flag
```

The MoE type is also named differently from every sibling (`GGUFQWenMoE` vs
`ModelWeights`), which is easy to trip over.

**Metadata keys are architecture-prefixed.** `llama.context_length` vs
`qwen3.context_length` vs `phi3.context_length`. A consumer that hardcodes the
`llama.` prefix silently falls back to a default context window for every other
family — no error, just a model that rejects prompts it should accept.

### Proposed API

```rust
pub trait QuantizedLm {
fn forward(&mut self, input: &Tensor, index_pos: usize) -> Result;
}

/// Select and construct the backend registered for `ct`'s `general.architecture`.
pub fn from_gguf(
ct: gguf_file::Content,
reader: &mut R,
device: &Device,
) -> Result>;
```

Every family already exposes `forward(&mut self, input, index_pos) -> Result`,
so the trait costs nothing to implement. The per-family extras (`dtype`,
`use_flash_attn`) can move into an options struct with sane defaults.

An accompanying `context_length(&Content) -> Option` that reads the right
namespace would remove the second footgun.

### Related

`quantized_qwen3_moe` cannot currently be used outside CUDA:
`candle_nn::moe_gemm_gguf` returns "only implemented for the cuda backend" on
CPU and Metal, and the prefill kernel accepts only F16/BF16 working dtypes. We
had to unwire that family after advertising it, because nothing surfaces those
constraints before the first expert layer runs. A dispatcher would be the
natural place to reject an unusable (family, device, dtype) combination at load.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the quantized_llama, quantized_qwen2, quantized_qwen3, quantized_qwen3_moe, quantized_gemma3, quantized_phi3, and other quantized_* modules, especially their from_gguf and forward signatures. Trace how each family reads architecture-prefixed metadata and identify the existing GGUF loading entry points. Done means a dispatcher and context_length API handle the registered families while surfacing unsupported device and dtype combinations.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.