microsoft / microsoft/onnxruntime

[CUDA] Qwen3.6-35B-A3B Throughput Optimization

Open
#28,987 2 comments 1 reaction 0 assignees View on GitHub
ep:CUDA stale
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

This is used to track the progress of Qwen3.6-35B-A3B Throughput Optimization.

Related PRs:
For Qwen 3.6:

* olive-recipes
[Add Qwen3.6-35B-A3B MoE VLM recipe (CUDA + CPU)](https://github.com/microsoft/olive-recipes/pull/492)

* onnxruntime-genai
[Fix CUDA QMoE INT4 export for Qwen3.5/3.6 MoE models](https://github.com/microsoft/onnxruntime-genai/pull/2209)
https://github.com/microsoft/onnxruntime-genai/pull/2218

* cuda op / kernels:
https://github.com/microsoft/onnxruntime/pull/28980
https://github.com/microsoft/onnxruntime/pull/28985
https://github.com/microsoft/onnxruntime/pull/28986 (Not needed if we have shared expert optimization like below)
https://github.com/microsoft/onnxruntime/pull/29028
https://github.com/microsoft/onnxruntime/pull/29038 (Need to extend to block quantization)
https://github.com/microsoft/onnxruntime/pull/29013
https://github.com/microsoft/onnxruntime/pull/29824
https://github.com/microsoft/onnxruntime/pull/29818

Related issues:
[AddExternalInitializers copies device (GPU) OrtValues per session instead of using them in place
](https://github.com/microsoft/onnxruntime/issues/29009)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the related olive-recipes, onnxruntime-genai, and ONNX Runtime PRs listed in the issue, including the CUDA op and kernel changes. Trace how those changes address Qwen3.6-35B-A3B throughput and determine the remaining work needed across the linked efforts. Done means the tracked optimization work is complete, but the issue does not define a specific benchmark or acceptance target.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, machine-learning
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.