mudler / mudler/vllm.cpp

perf(BACKEND-GATE-ROCM-SGLANG): close the Qwen3-4B Strix c4 gap

Open
#3,076 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-GATE-ROCM-SGLANG

Owner: current Strix campaign operator. Parent: #3053. Spec: .agents/specs/strix-qwen3-4b-c4-performance.md (commit before implementation).

User approved measurement-first work to bring vllm.cpp at least level with production vLLM at concurrency four on Strix. Two diagnostic qualification corpora gave medians 61.5947184118 versus 79.5022690204 output tokens/s. Correctness failed, so these are NOT_ACCEPTED diagnostic observations, not a benchmark or accepted ratio. Full record: #3053 issuecomment-5586767582.

Keep Qwen3-4B BF16 revision 1cfa9a7208912126459214e8b04321603b3df60c, six raw prompts, 128 greedy tokens, c1/c4, pinned engine baseline6e3cbfb940be89e28d1d71c264fd8c3a4e44afeb and production vLLMe126687a9a828d513c01a07cd69f025f27d63280. Obtain complete same-tool HIP/API/kernel traces before attribution. Trace06 is incomplete and cannot satisfy parity. Source candidates: rocm_embedding.hip127-142 per-step allocation/synchronization/copy/free; rocm.cpp91-97 and qwen3.cpp1184-1186 reject dense graph replay; paged attention BF16/head128/GQA4 does not use the fused-GQA arm. These are unmeasured hypotheses.

Coordinate existing graph #332/PR#2777 and async mirror PR#2779 before duplicating work. #3043 owns the separate 27B workload. Implement only a measured, individually specified lever, preserving invalid-input errors and production reachability. Require smallest red-before public-entry regression, focused and full gates, independent static and mutation review, operator verification, and a same-binary A/B on an idle leased host. Correctness requires the declared exact or separately ratified distributional gate. No eager denominator, candidate-driven tolerance, pin substitution, or ceiling claim. Final acceptance requires matched warmed repeated throughput at c4 >= vLLM, with c1, latency and memory obligations reported; otherwise keep the gap open and name the next traceable hypothesis.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read .agents/specs/strix-qwen3-4b-c4-performance.md and the full record in #3053 first, then reproduce the pinned Qwen3-4B BF16 c1/c4 measurements on an idle Strix host. Collect complete same-tool HIP/API/kernel traces before evaluating the cited rocm_embedding.hip, rocm.cpp, and qwen3.cpp locations. Done means an individually measured lever passes the required regression and gates, with matched warmed c4 throughput, c1, latency, memory, and correctness reported.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.