huggingface / huggingface/candle
`quantized_lm`: dispatch Qwen3.5 GGUF checkpoints
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
## Problem
`quantized_qwen3_5::ModelWeights` loads a Qwen3.5 GGUF — the hybrid that
interleaves Gated DeltaNet linear-attention layers with full-attention ones —
but `quantized_lm::Architecture` has no arm for it. Its `from_gguf_architecture`
stops at `qwen3moe`, so a caller holding a Qwen3.5 GGUF cannot reach the backend
that reads it: either it names the family itself, which is the per-family match
this module exists to delete, or the checkpoint is refused as an unrecognized
architecture.
`qwen3moe` is not a substitute. It resolves to `quantized_qwen3_moe`, a
mixture-of-experts model with full attention throughout — a different
architecture, not a spelling of the same one. A Qwen3.5 checkpoint routed there
would load against the wrong layer structure.
The gap is easy to miss precisely because the backend exists: nothing in the
crate is missing except the entry that makes it reachable.
## What the fix needs
1. **An `Architecture` variant and its `general.architecture` spelling**, so
`from_gguf` resolves the family the same way every other backend is
resolved.
2. **A `QuantizedLm` impl.** `ModelWeights::forward(&mut self, input, offset)`
already has the trait's exact shape — `forward(&mut self, input, index_pos)`
— and the type is `Send`, so the impl is a forwarding one-liner.
3. **Nothing else.** `clear_kv_cache` is already there for callers that reset
between requests; the linear-attention state it clears is the model's own.
## Proposed API
```rust
pub enum Architecture {
// …
Qwen3Moe,
Qwen3_5,
}
// in `from_gguf_architecture`
"qwen3_5" => Self::Qwen3_5,
```
with the usual forwarding impl beside the other families:
```rust
impl QuantizedLm for quantized_qwen3_5::ModelWeights {
fn forward(&mut self, input: &Tensor, index_pos: usize) -> Result {
Self::forward(self, input, index_pos)
}
}
```
The `general.architecture` spelling is the one open question. `quantized_qwen3_5`
does not read the key itself, and the example loads the file directly, so the
string is whatever the conversion tooling writes — worth confirming against a
real checkpoint before the arm is added, since an arm keyed on the wrong
spelling is as unreachable as no arm at all.
## Note
This is the same class of gap as #3783: a backend present in the crate but
absent from the dispatcher is dead code, which is what had happened to `glm4`,
`lfm2`, `phi2` and `qwen3moe`. Qwen3.5 is the newest instance, arriving after
the dispatcher was written.
I am happy to submit the PR once the architecture string is confirmed. Related:
#3783, #3794.
Contributor guide
No contributing guide indexed for this repository
Research direction
Inspect quantized_lm::Architecture and from_gguf_architecture, then compare the existing family dispatch arms with quantized_qwen3_5::ModelWeights and its forward signature. Confirm the general.architecture spelling against a real Qwen3.5 GGUF checkpoint before adding the variant and forwarding implementation. Done means a Qwen3.5 checkpoint resolves through the dispatcher and reaches the existing backend.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100