mudler / mudler/vllm.cpp

perf(ROCm): parallel random sample with shared primitives

Open
#2,775 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: GFX1100-TG200

Replaces the serial <<<n,1>>> random-sample kernel with a block-cooperative argmax reduction over Gumbel scores, mirroring the CUDA fix from #1984. The RNG and argmax reduce come from include/vt/sample_common.h, shared across CPU/CUDA/ROCm so bit-identity is a build property, not a copy-sync property. The serial path is retained behind VT_FAST_RANDOM_SAMPLE=0 for same-binary A/B gating.

Adds ROCm + CUDA subprocess A/B test cases that re-exec the binary with VT_FAST_RANDOM_SAMPLE=0 and =1 and assert byte-identical token ids across widths up to 248320 (Qwen3.8-27B vocab).

Verified: test_sampler 21/21 pass, 114 assertions. Token-identical to upstream baseline on Qwen3.5-4B Q4_K.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with include/vt/sample_common.h and locate the random-sample kernel and test_sampler entry points. Compare the serial and block-cooperative paths, then run the ROCm and CUDA subprocess A/B cases with VT_FAST_RANDOM_SAMPLE=0 and =1. Done means test_sampler passes and token IDs remain byte-identical across the stated vocabulary widths.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.