huggingface / huggingface/candle

`quantized`: expose whether a `GgmlDType` supports matmul on a given device

Open
#3,781 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21.1k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

### Problem

There is no way to ask, ahead of loading a checkpoint, whether a given
`GgmlDType` can actually be multiplied on a given `Device`. The answer exists
only inside kernel dispatch, as the fallthrough arm of several `match`
statements:

- `candle-core/src/quantized/cuda.rs` — `dequantize_mul_mat_vec`,
`mul_mat_vec_via_q8_1` and `mul_mat_via_q8_1` each end in
`_ => crate::bail!("unsupported dtype for quantized matmul {dtype:?}")`.
- `candle-core/src/quantized/metal.rs` — covers every `GgmlDType`.

The sets genuinely differ, and not in an obvious direction:

| | CUDA | Metal |
|---|---|---|
| `Q4_0` `Q4_1` `Q5_0` `Q5_1` `Q8_0` `Q2K` `Q3K` `Q4K` `Q5K` `Q6K` | ✅ | ✅ |
| `Q8_1`, `Q8K` | dequantize only, **no matmul** | ✅ |

`Q8_1`/`Q8K` are the trap: they have CUDA *dequantize* kernels, so a checkpoint
carrying them loads and occupies VRAM, then fails at the first decode step with
a generic dtype error.

### Proposed API

```rust
impl GgmlDType {
/// Whether a quantized matmul kernel exists for this block type on `device`.
pub fn supports_matmul(self, device: &Device) -> bool;
}
```

Implemented next to the dispatch it describes, so the two cannot drift.

`F32`/`F16`/`BF16` report `true` on any device: `QMatMul::from_arc` dequantizes
them into a dense `Tensor` rather than dispatching to a quantized kernel, so
they never reach the matmul match at all — another piece of knowledge currently
only obtainable by reading the source.

There is deliberately no `supports_dequantize` counterpart: it would return
`true` for every dtype on every backend by construction, since the CUDA path
falls back to host-side dequantization for anything without a fast kernel. The
question worth asking there is a *performance* one ("is there a device-native
dequantize kernel"), which is not what this API answers and should not share
its name.

### Concrete caller

`astorise/tachyon-mesh` (`core-host/src/ai_inference/candle_llm_runtime.rs`)
validates a GGUF checkpoint's block types *before* uploading weights, so an
unrunnable model fails at load with an actionable message instead of at the
first decode step with VRAM already claimed:

```
GGUF tensor `blk.0.ffn_down.weight` uses block type `Q8K`, which has no CUDA
quantized-matmul kernel (2 of 291 tensors affected); requantize to a K-quant or
legacy Q4/Q5/Q8 type, or bind this model to `cpu`
```

Today that check is a hardcoded `matches!` whose doc comment admits it "mirrors
the dtype matches in `candle-core/src/quantized/cuda.rs`" — a duplicated
invariant across a pinned tag, which goes stale silently rather than failing
when the kernel set changes upstream.

The per-device signature is what makes this an API rather than a constant: the
sets differ between backends (`Q8_1`/`Q8K` work on Metal, not on CUDA), so a
serving layer targeting both cannot hold a single table. That crate will drop
its copy and call `supports_matmul` once this lands.

No in-tree caller today — this is an API for downstream consumers, exposing a
fact that currently exists only as the fallthrough arm of three private `match`
statements.

### Related

`candle_nn::moe_gemm_gguf` has the same shape of problem one layer up: it is
CUDA-only and accepts a narrower weight dtype set (`Q2K`–`Q6K`, `Q8_0`) than the
generic matmul path, with no way to query either fact. A checkpoint whose expert
tensors use `Q4_0` passes every available check and then fails at its first
expert layer.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the quantized dispatch functions in candle-core/src/quantized/cuda.rs—dequantize_mul_mat_vec, mul_mat_vec_via_q8_1, and mul_mat_via_q8_1—and the corresponding coverage in candle-core/src/quantized/metal.rs. Add GgmlDType::supports_matmul for the device-specific kernel sets, including dense F32/F16/BF16 handling, and verify that the reported support matches dispatch behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.