huggingface / huggingface/candle
Add the gpt-oss model (OpenAI open-weights MoE) to candle-transformers
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
## Summary
OpenAI's gpt-oss family (gpt-oss-20b / gpt-oss-120b, Apache-2.0) is a sparse MoE with attention sinks, alternating sliding-window / full attention, YaRN RoPE, and MXFP4-quantized experts. `candle-transformers` has no gpt-oss model today, and users have asked about running these weights (#3047).
I have a working, numerically-verified implementation and would like to contribute it. Opening this issue first to coordinate, given the prior art below.
## Prior art
- #3129 ("Implement Gpt oss") - closed, was not functional.
- #3391 ("Mxfp4 gpt oss implementation") - open since 2026-03 with no reviews; takes the heavy path (a native MXFP4 quantized type in `candle-core` plus new CUDA/Metal kernels).
## Proposed approach (lighter)
Dequantize the MXFP4 experts to the model dtype at load - the experts are the only MXFP4 tensors; everything else is bf16 - so the change is confined to one model file plus a small dequant helper: no `candle-core` changes, no new kernels. Native MXFP4 inference and a `RotatingKvCache` for the sliding layers are natural follow-ups.
## Verification
Logits match HF transformers `modeling_gpt_oss` on gpt-oss-20b in bf16: argmax + top-4 exact, cosine ~0.9996, residual at the bf16 representability floor (full numbers in the PR). The example generates coherent text (~22 tok/s on an A6000, bf16).
Happy to align on direction (dequant-first vs native MXFP4) before or during review. PR to follow.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.