huggingface / huggingface/candle

`quantized_lm`: dispatch Qwen3.5 GGUF checkpoints

Open
#3,820 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

## Problem

`quantized_qwen3_5::ModelWeights` loads a Qwen3.5 GGUF — the hybrid that
interleaves Gated DeltaNet linear-attention layers with full-attention ones —
but `quantized_lm::Architecture` has no arm for it. Its `from_gguf_architecture`
stops at `qwen3moe`, so a caller holding a Qwen3.5 GGUF cannot reach the backend
that reads it: either it names the family itself, which is the per-family match
this module exists to delete, or the checkpoint is refused as an unrecognized
architecture.

`qwen3moe` is not a substitute. It resolves to `quantized_qwen3_moe`, a
mixture-of-experts model with full attention throughout — a different
architecture, not a spelling of the same one. A Qwen3.5 checkpoint routed there
would load against the wrong layer structure.

The gap is easy to miss precisely because the backend exists: nothing in the
crate is missing except the entry that makes it reachable.

## What the fix needs

1. **An `Architecture` variant and its `general.architecture` spelling**, so
`from_gguf` resolves the family the same way every other backend is
resolved.
2. **A `QuantizedLm` impl.** `ModelWeights::forward(&mut self, input, offset)`
already has the trait's exact shape — `forward(&mut self, input, index_pos)`
— and the type is `Send`, so the impl is a forwarding one-liner.
3. **Nothing else.** `clear_kv_cache` is already there for callers that reset
between requests; the linear-attention state it clears is the model's own.

## Proposed API

```rust
pub enum Architecture {
// …
Qwen3Moe,
Qwen3_5,
}

// in `from_gguf_architecture`
"qwen3_5" => Self::Qwen3_5,
```

with the usual forwarding impl beside the other families:

```rust
impl QuantizedLm for quantized_qwen3_5::ModelWeights {
fn forward(&mut self, input: &Tensor, index_pos: usize) -> Result {
Self::forward(self, input, index_pos)
}
}
```

The `general.architecture` spelling is the one open question. `quantized_qwen3_5`
does not read the key itself, and the example loads the file directly, so the
string is whatever the conversion tooling writes — worth confirming against a
real checkpoint before the arm is added, since an arm keyed on the wrong
spelling is as unreachable as no arm at all.

## Note

This is the same class of gap as #3783: a backend present in the crate but
absent from the dispatcher is dead code, which is what had happened to `glm4`,
`lfm2`, `phi2` and `qwen3moe`. Qwen3.5 is the newest instance, arriving after
the dispatcher was written.

I am happy to submit the PR once the architecture string is confirmed. Related:
#3783, #3794.

Contributor guide

No contributing guide indexed for this repository

Research direction

Inspect quantized_lm::Architecture and from_gguf_architecture, then compare the existing family dispatch arms with quantized_qwen3_5::ModelWeights and its forward signature. Confirm the general.architecture spelling against a real Qwen3.5 GGUF checkpoint before adding the variant and forwarding implementation. Done means a Qwen3.5 checkpoint resolves through the dispatcher and reaches the existing backend.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.