huggingface / huggingface/candle
ModelOpt checkpoints: the weight source stops at the projection; detection, shards and config are still on every consumer
- Dominant language
- Rust
- Stars
- 21.1k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
## Summary
#3846 asked for a ModelOpt weight source. What landed covers the *projection*: given a `VarBuilder` already positioned at a linear layer, `quantized_nvfp4::auto_linear` reads the packed weight, the block scales and the tensor scale, and picks the operator. That is the hard half, and it works.
The other half is still on every consumer: deciding that a directory *is* a ModelOpt NVFP4 checkpoint, and reading the config it ships.
Concretely, before `auto_linear` can be called even once, a caller must:
- recognise NVFP4 from `W4A16_NVFP4` metadata in `config.json` or `hf_quant_config.json`, or from ModelOpt tensor names such as `.weight_scale_2`;
- read `model.safetensors.index.json`, resolve the shard set, and hand the right paths to `VarBuilder::from_mmaped_safetensors`;
- deserialize the checkpoint's own `config.json` into the model's `Config`.
None of that is application logic, and all of it is identical for every consumer.
## The sharpest symptom: two config types for one model
`qwen3_5::Config` and `TextConfig` both derive `serde::Deserialize`, so they look ready to read a checkpoint. They are not, for a ModelOpt export: the nesting and several field names differ from what the file carries. So a downstream ends up with a *second* config type that deserializes the real file, plus a translation function into candle's.
That is what we have. The translation is 35 lines of field-by-field copying whose only reason to exist is that the two shapes disagree:
```rust
pub(crate) fn upstream_config(config: &Qwen35MoeConfig) -> UpstreamConfig {
UpstreamConfig { text_config: TextConfig {
vocab_size: config.vocab_size,
head_dim: Some(config.head_dim),
hidden_act: candle_nn::Activation::Silu, // not in the file
attention_bias: false, // not in the file
// … 20 more
}}
}
```
Two of those fields are not in the checkpoint at all and have to be pinned by the caller from knowledge of the architecture — which is precisely the knowledge candle has and the caller should not need.
Any second consumer will write this same struct and this same function, and the two copies will disagree the first time a field is added upstream.
## What this asks for
A directory-level entry point beside the projection-level one. Roughly:
```rust
// Detects the format and reads what the directory states about itself.
let ckpt = ModelOptCheckpoint::open(path)?; // None if not ModelOpt/NVFP4
let cfg: qwen3_5::Config = ckpt.model_config()?; // the file's own config.json
let vb = ckpt.var_builder(dtype, &device)?; // index + shards resolved
let model = qwen3_5::ModelForCausalLM::new_with_projections(
&cfg, vb,
|name, vb, in_dim, out_dim, bias| {
Ok(Box::new(auto_linear(in_dim, out_dim, bias, vb.pp(name), cfg)?))
},
)?;
```
The shape matters more than the names. Three things are being asked for, and they are separable:
1. **Detection** — is this directory a ModelOpt NVFP4 checkpoint? Today every consumer re-derives it from metadata keys and tensor-name suffixes.
2. **Shard resolution** — index to shard paths. Mechanical, and identical everywhere.
3. **Config deserialization** — the checkpoint's `config.json` into the model's `Config`, including the fields the file does not carry.
(3) is the one that removes a whole parallel type from downstreams. (1) and (2) are smaller but are pure duplication.
## Why this is candle's and not the application's
The line we have been drawing: candle owns the model, the kernels and the checkpoint format; the application owns policy. Detection, shard resolution and config parsing are all format facts. What legitimately stays with a consumer is which producer versions it has qualified, what it does when a checkpoint fails validation, and where it will run — none of which candle should decide.
Our own `modelopt_nvfp4.rs` is ~2.6k lines. Most of it is (1), (2) and the tensor grouping that `auto_linear` already made redundant; the part that is genuinely ours — a fail-closed compatibility profile that refuses a producer version we have not qualified — is a few hundred lines and belongs where it is.
## Notes from having written it downstream
- Detection has to accept both spellings of the packed weight (`weight` for ModelOpt, `weight_packed` for compressed-tensors) — the same distinction #3846 settled at the projection level applies to the directory-level probe.
- `hf_quant_config.json` is not always present; some exports carry the quantization block inside `config.json`. Both are real.
- A checkpoint is mixed: dense BF16, FP8 and W4A16 NVFP4 tensors in one graph. Detection must not require that every linear be NVFP4.
- Sharded and single-file checkpoints both occur; `model.safetensors.index.json` is absent in the second case.
- Fields like `hidden_act` and `attention_bias` are architecture facts rather than file contents. If `Config` deserialization owns them via `#[serde(default = …)]`, no consumer has to guess them.
## Context
Follows #3846 (projection-level weight source) and #3857 (packed CPU operator). Downstream in [Tachyon-Mesh](https://github.com/astorise/Tachyon-Mesh) we have just deleted a local reimplementation of Qwen 3.5 in favour of candle's; what remains model-specific on our side is almost entirely this loading gap and the policy around it.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with quantized_nvfp4::auto_linear and the existing modelopt_nvfp4.rs implementation, then inspect how config.json, hf_quant_config.json, and model.safetensors.index.json are currently handled. Define the directory-level checkpoint entry point around detection, shard resolution, and config deserialization; done means single-file and sharded ModelOpt NVFP4 checkpoints are recognized without consumer-side duplication.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100