huggingface / huggingface/candle
qwen3_5: a projection seam, so a non-GGUF quantized checkpoint can reuse the layers
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
## Problem
`qwen3_5.rs` builds its projections concretely:
```rust
pub struct Qwen3_5GatedDeltaNet {
// …
in_proj_qkv: Linear,
in_proj_z: Linear,
in_proj_b: Linear,
in_proj_a: Linear,
out_proj: Linear,
// …
}
impl Qwen3_5GatedDeltaNet {
pub fn new(cfg: &Config, vb: VarBuilder) -> Result {
let in_proj_qkv = linear_no_bias(hidden_size, conv_dim, vb.pp("in_proj_qkv"))?;
// …
}
}
```
`Qwen3_5Attention` and `Qwen3_5MLP` are the same. There are exactly two ways to
get a Qwen 3.5 through this crate: dense weights via `qwen3_5.rs`, or GGUF via
`quantized_qwen3_5.rs`. A checkpoint in a third format has no way in.
That third format is not hypothetical. NVIDIA's ModelOpt publishes Qwen 3.5
checkpoints as NVFP4 safetensors — packed E2M1 nibbles with per-block E4M3
scales — and the tensor names, the config, the layer topology and every
non-projection operation are byte-for-byte the ones `qwen3_5.rs` already
implements. The *only* thing that differs is how a `[rows, cols]` weight is
multiplied.
So a caller with such a checkpoint reimplements the whole file — the conv1d
short convolution, the GQA repeat, the L2 normalisation of q and k, the decay
and beta derivation, the gated RMSNorm, the whole `Qwen3_5DecoderLayer`
structure — to change one line of arithmetic. We did, and it is roughly a
thousand lines of duplicated model semantics whose only job is to be identical
to this file, including the risk that it silently stops being.
This is also what keeps `gated_delta_rule_chunked` out of reach for those
callers. That module is the good part — an O(T/chunk) prefill scan with a
sequential reference test — and a reimplementation cannot call it, because
everything around it that feeds it (conv, projections, head layout) is inside
the layer they could not reuse.
## Why the same seam as #3809
`llama.rs` had this exact problem and solved it: `BlockMlp` plus
`load_with_mlp_factory` let a caller substitute the feed-forward computation
while keeping every other line of the reference implementation. #3828 did the
same for the reduction in a tensor-parallel linear. The pattern is established
here, and this is the same shape of request: the model knows the algorithm, the
caller knows how its weights multiply.
The difference is only which seam. For Qwen 3.5 the substitution point is the
projection, because that is where a quantization format lives.
## Proposed API
```rust
/// A linear projection inside a Qwen 3.5 layer.
pub trait Projection: Send + Sync {
fn forward(&self, xs: &Tensor) -> Result;
}
/// Build one projection. `name` is the checkpoint-relative name
/// (`in_proj_qkv`, `q_proj`, `down_proj`, …) and `vb` is rooted at the layer,
/// so a factory can find its own weights however its format stores them.
pub type ProjectionFactory<'a> =
&'a dyn Fn(&str, &VarBuilder, usize, usize) -> Result>;
impl Model {
pub fn new_with_projections(
cfg: &Config,
vb: VarBuilder,
projections: ProjectionFactory<'_>,
) -> Result;
}
```
`Model::new` then becomes `new_with_projections` with a factory returning
`linear_no_bias`, which is both the default and the proof that the seam did not
change dense behaviour.
Blanket `impl Projection for T where T: Module` would let any existing layer
type — `Linear`, `QMatMul` wrappers, an NVFP4 operator — be passed without a
newtype.
## Note
This depends on the sibling issue asking `candle-nvfp4-kernels` #3831 for a
device-resident `Tensor -> Tensor` operator: without one there is nothing to
put through the seam, since a host-in/host-out kernel cannot implement
`Projection` meaningfully. The two are worth reading together, and this one
lands second.
I am happy to submit the PR. The test I would bring with it is the one that
made this worth doing: a factory returning `linear_no_bias` must produce output
identical to `Model::new` on the same checkpoint, so the seam is provably free.
Related: #3809, #3828, #3824, #3831.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in qwen3_5.rs and compare the projection-factory pattern in llama.rs from #3809, along with the reduction seam from #3828. Trace Model::new, Qwen3_5GatedDeltaNet, Qwen3_5Attention, and Qwen3_5MLP, then run the existing Qwen tests. Done means a default linear_no_bias factory preserves Model::new output and the projection seam can support alternate weight multiplication.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100