huggingface / huggingface/candle
`candle-transformers`: dispatch GGUF checkpoints by `general.architecture`
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
### Problem
`candle-transformers` ships a growing set of `quantized_*` modules —
`quantized_llama`, `quantized_qwen2`, `quantized_qwen3`, `quantized_qwen3_moe`,
`quantized_gemma3`, `quantized_phi3`, `quantized_glm4`, … — but no way to load
"whatever GGUF this is". Every consumer writes the same dispatcher.
Two things make that dispatcher more annoying than it should be:
**Signatures diverge.** They are close enough to look uniform and different
enough to need a wrapper enum:
```rust
quantized_llama::ModelWeights::from_gguf(ct, reader, device)
quantized_qwen2::ModelWeights::from_gguf(ct, reader, device)
quantized_qwen3::ModelWeights::from_gguf(ct, reader, device)
quantized_gemma3::ModelWeights::from_gguf(ct, reader, device)
quantized_qwen3_moe::GGUFQWenMoE::from_gguf(ct, reader, device, dtype) // + dtype
quantized_phi3::ModelWeights::from_gguf(use_flash_attn, ct, reader, device) // leading flag
```
The MoE type is also named differently from every sibling (`GGUFQWenMoE` vs
`ModelWeights`), which is easy to trip over.
**Metadata keys are architecture-prefixed.** `llama.context_length` vs
`qwen3.context_length` vs `phi3.context_length`. A consumer that hardcodes the
`llama.` prefix silently falls back to a default context window for every other
family — no error, just a model that rejects prompts it should accept.
### Proposed API
```rust
pub trait QuantizedLm {
fn forward(&mut self, input: &Tensor, index_pos: usize) -> Result;
}
/// Select and construct the backend registered for `ct`'s `general.architecture`.
pub fn from_gguf(
ct: gguf_file::Content,
reader: &mut R,
device: &Device,
) -> Result>;
```
Every family already exposes `forward(&mut self, input, index_pos) -> Result`,
so the trait costs nothing to implement. The per-family extras (`dtype`,
`use_flash_attn`) can move into an options struct with sane defaults.
An accompanying `context_length(&Content) -> Option` that reads the right
namespace would remove the second footgun.
### Related
`quantized_qwen3_moe` cannot currently be used outside CUDA:
`candle_nn::moe_gemm_gguf` returns "only implemented for the cuda backend" on
CPU and Metal, and the prefill kernel accepts only F16/BF16 working dtypes. We
had to unwire that family after advertising it, because nothing surfaces those
constraints before the first expert layer runs. A dispatcher would be the
natural place to reject an unusable (family, device, dtype) combination at load.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the quantized_llama, quantized_qwen2, quantized_qwen3, quantized_qwen3_moe, quantized_gemma3, quantized_phi3, and other quantized_* modules, especially their from_gguf and forward signatures. Trace how each family reads architecture-prefixed metadata and identify the existing GGUF loading entry points. Done means a dispatcher and context_length API handle the registered families while surfacing unsupported device and dtype combinations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100