huggingface / huggingface/candle
Contribute Qwen3.5/3.6 35B-A3B port + 3 fused perf kernels — packaging guidance?
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
**TL;DR**
I have a complete Qwen3.6-35B-A3B port (Hybrid Gated DeltaNet + 256/9 MoE) plus 3 byte-match-validated NVRTC perf kernels (cumulative ~+15.7% decode on RTX 3060 Ti) ready to upstream. Looking for packaging guidance before submitting — one big PR, or split? — to avoid burning reviewer time on the wrong shape.
## Context
Qwen3.6 35B-A3B is a different architecture from upstream's `quantized_qwen3_moe.rs`:
- **Hybrid attention**: 30 Gated DeltaNet recurrent layers + 10 FullAttn (4:1 interval), not pure attention.
- **MoE routing**: 256 experts + 1 shared (sigmoid-gated), top-8 per token, vs Qwen3-MoE's 128/8 / no shared.
- **Routing kernel**: needs custom softmax + top-K + renormalize for ggml clamp parity (validated against ggml CPU oracle).
- **DeltaNet AR**: state-coupled per-token recurrence (`conv_state` shift + L2-normed q/k + gated norm); conv1d K=4 depthwise; state lives across decode steps.
The port lives in `candle-transformers/src/models/quantized_qwen3_5_moe.rs` (~4200 LOC). Phase A.6 validation: argmax 16/16 + first-layer attn_residual ~1e-6 against ggml CPU oracle. (PR #3476 — uninit `fast_mmvq` workspace — is from this same work series; still open, no urgency.)
## Perf kernels added in candle-core
Three byte-match-validated kernels with measured wins on the production hot path. All have argmax 16/16 byte-match + 6-run median compare + opt-out env gate:
| Kernel | Lines | A/B decode delta | Mechanism |
|---|---|---|---|
| `sigmoid_gate_combine_f32` | ~300 | **+4.8%** | 5 candle ops (sigmoid + 2× cast + broadcast_mul + add) → 1 NVRTC launch. Sigmoid-gated shared expert combine. |
| `QCudaStorage::uninit` (unsafe) | ~50 | **+2.8%** | Skip `cudaMemsetAsync` on a compact MoE expert buffer that the caller exhaustively overwrites via D2D + HtoD before any read. |
| `delta_net_conv1d_silu_state_t1` | ~290 | **+9.8%** | 8 candle ops (narrow + cat + transpose + contiguous + mul + sum + silu + state shift) → 1 NVRTC launch for T=1 decode path on 30 DeltaNet layers. |
Cumulative on Qwen3.6 35B-A3B Q4_K_M / RTX 3060 Ti / Win11: **11.75 → 13.59 t/s = +15.7%** decode p50 (median-of-3 runs per A/B). llama.cpp on the same hardware is 19.6 t/s; this closes the gap from -40% to -31%.
Each kernel:
- argmax byte-match against legacy candle-op chain (logits within fp32 fast-math noise floor)
- 6-run median perf compare with `CANDLE_QWEN36_LEGACY_*` env opt-out
- Independent commit; cleanly cherry-pickable
## Question
Three plausible packagings — would prefer your guidance before sinking PR-prep time:
1. **One PR**: model port + 3 kernels in candle-core, single ~4500-line submission. Cleanest "Add Qwen3.6 35B-A3B with optimized inference kernels" narrative; largest review surface.
2. **Two PRs**:
1. Model port (correctness only — same kernel use as today's candle ops)
2. Follow-up with kernel optimizations once (1) lands
Smaller per-PR; perf reviewable independently against the just-landed model.
3. **Three+ PRs**: model port + each kernel separately. Smallest per-PR but 4 review threads for one logical contribution.
Open to other framings. Examples:
- 1.b: model port + the two safe-API kernels (`sigmoid_gate_combine_f32` + `delta_net_conv1d_silu_state_t1`) in one PR, defer the unsafe `QCudaStorage::uninit` to a follow-up where the unsafe contract can be discussed independently.
- 2.b: open a 3rd Q-K MoE model entirely (not Qwen-named) so the kernels have a clear upstream user without bringing the Qwen3.6 architecture in immediately.
## What I'm NOT asking for
Just a quick read on shape preference. Not asking for design-review of any specific kernel here — those discussions belong on each PR. Also not in a hurry; happy to wait until you have queue capacity.
Thanks for any guidance.
---
Contributor guide
No contributing guide indexed for this repository
Research direction
Review candle-transformers/src/models/quantized_qwen3_5_moe.rs and the three kernel areas described in candle-core, then compare the proposed one-, two-, and three-PR packaging options. Done means maintainers have given clear guidance on the contribution shape; the issue does not define an implementation task or a specific test target.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100