huggingface / huggingface/candle

Contribute Qwen3.5/3.6 35B-A3B port + 3 fused perf kernels — packaging guidance?

Open
#3,514 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

**TL;DR**

I have a complete Qwen3.6-35B-A3B port (Hybrid Gated DeltaNet + 256/9 MoE) plus 3 byte-match-validated NVRTC perf kernels (cumulative ~+15.7% decode on RTX 3060 Ti) ready to upstream. Looking for packaging guidance before submitting — one big PR, or split? — to avoid burning reviewer time on the wrong shape.

## Context

Qwen3.6 35B-A3B is a different architecture from upstream's `quantized_qwen3_moe.rs`:

- **Hybrid attention**: 30 Gated DeltaNet recurrent layers + 10 FullAttn (4:1 interval), not pure attention.
- **MoE routing**: 256 experts + 1 shared (sigmoid-gated), top-8 per token, vs Qwen3-MoE's 128/8 / no shared.
- **Routing kernel**: needs custom softmax + top-K + renormalize for ggml clamp parity (validated against ggml CPU oracle).
- **DeltaNet AR**: state-coupled per-token recurrence (`conv_state` shift + L2-normed q/k + gated norm); conv1d K=4 depthwise; state lives across decode steps.

The port lives in `candle-transformers/src/models/quantized_qwen3_5_moe.rs` (~4200 LOC). Phase A.6 validation: argmax 16/16 + first-layer attn_residual ~1e-6 against ggml CPU oracle. (PR #3476 — uninit `fast_mmvq` workspace — is from this same work series; still open, no urgency.)

## Perf kernels added in candle-core

Three byte-match-validated kernels with measured wins on the production hot path. All have argmax 16/16 byte-match + 6-run median compare + opt-out env gate:

| Kernel | Lines | A/B decode delta | Mechanism |
|---|---|---|---|
| `sigmoid_gate_combine_f32` | ~300 | **+4.8%** | 5 candle ops (sigmoid + 2× cast + broadcast_mul + add) → 1 NVRTC launch. Sigmoid-gated shared expert combine. |
| `QCudaStorage::uninit` (unsafe) | ~50 | **+2.8%** | Skip `cudaMemsetAsync` on a compact MoE expert buffer that the caller exhaustively overwrites via D2D + HtoD before any read. |
| `delta_net_conv1d_silu_state_t1` | ~290 | **+9.8%** | 8 candle ops (narrow + cat + transpose + contiguous + mul + sum + silu + state shift) → 1 NVRTC launch for T=1 decode path on 30 DeltaNet layers. |

Cumulative on Qwen3.6 35B-A3B Q4_K_M / RTX 3060 Ti / Win11: **11.75 → 13.59 t/s = +15.7%** decode p50 (median-of-3 runs per A/B). llama.cpp on the same hardware is 19.6 t/s; this closes the gap from -40% to -31%.

Each kernel:
- argmax byte-match against legacy candle-op chain (logits within fp32 fast-math noise floor)
- 6-run median perf compare with `CANDLE_QWEN36_LEGACY_*` env opt-out
- Independent commit; cleanly cherry-pickable

## Question

Three plausible packagings — would prefer your guidance before sinking PR-prep time:

1. **One PR**: model port + 3 kernels in candle-core, single ~4500-line submission. Cleanest "Add Qwen3.6 35B-A3B with optimized inference kernels" narrative; largest review surface.

2. **Two PRs**:
1. Model port (correctness only — same kernel use as today's candle ops)
2. Follow-up with kernel optimizations once (1) lands
Smaller per-PR; perf reviewable independently against the just-landed model.

3. **Three+ PRs**: model port + each kernel separately. Smallest per-PR but 4 review threads for one logical contribution.

Open to other framings. Examples:

- 1.b: model port + the two safe-API kernels (`sigmoid_gate_combine_f32` + `delta_net_conv1d_silu_state_t1`) in one PR, defer the unsafe `QCudaStorage::uninit` to a follow-up where the unsafe contract can be discussed independently.
- 2.b: open a 3rd Q-K MoE model entirely (not Qwen-named) so the kernels have a clear upstream user without bringing the Qwen3.6 architecture in immediately.

## What I'm NOT asking for

Just a quick read on shape preference. Not asking for design-review of any specific kernel here — those discussions belong on each PR. Also not in a hurry; happy to wait until you have queue capacity.

Thanks for any guidance.

---

Contributor guide

No contributing guide indexed for this repository

Research direction

Review candle-transformers/src/models/quantized_qwen3_5_moe.rs and the three kernel areas described in candle-core, then compare the proposed one-, two-, and three-PR packaging options. Done means maintainers have given clear guidance on the contribution shape; the issue does not define an implementation task or a specific test target.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.