huggingface / huggingface/candle

Add the gpt-oss model (OpenAI open-weights MoE) to candle-transformers

Open
#3,580 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

## Summary

OpenAI's gpt-oss family (gpt-oss-20b / gpt-oss-120b, Apache-2.0) is a sparse MoE with attention sinks, alternating sliding-window / full attention, YaRN RoPE, and MXFP4-quantized experts. `candle-transformers` has no gpt-oss model today, and users have asked about running these weights (#3047).

I have a working, numerically-verified implementation and would like to contribute it. Opening this issue first to coordinate, given the prior art below.

## Prior art

- #3129 ("Implement Gpt oss") - closed, was not functional.
- #3391 ("Mxfp4 gpt oss implementation") - open since 2026-03 with no reviews; takes the heavy path (a native MXFP4 quantized type in `candle-core` plus new CUDA/Metal kernels).

## Proposed approach (lighter)

Dequantize the MXFP4 experts to the model dtype at load - the experts are the only MXFP4 tensors; everything else is bf16 - so the change is confined to one model file plus a small dequant helper: no `candle-core` changes, no new kernels. Native MXFP4 inference and a `RotatingKvCache` for the sliding layers are natural follow-ups.

## Verification

Logits match HF transformers `modeling_gpt_oss` on gpt-oss-20b in bf16: argmax + top-4 exact, cosine ~0.9996, residual at the bf16 representability floor (full numbers in the PR). The example generates coherent text (~22 tok/s on an A6000, bf16).

Happy to align on direction (dequant-first vs native MXFP4) before or during review. PR to follow.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.