mudler / mudler/vllm.cpp

Port async device-mirror combine/scatter kernels to ROCm

Open
#2,776 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-ROCM

The async device-mirror path eliminates the D2H sampled-token round trip by keeping last_sampled_tokens device-resident and splicing them into input_ids on the GPU. The current ROCm backend does not advertise this capability, so the shared runner does not reach the ROCm combine and scatter path.

Port the CUDA combine, scatter, and apply kernels to HIP in a current-main change. Route them through the shared runner without pulling CUDA headers into HIP-only builds. Add a production server or C API correctness case with non-speculative admission, compaction or swap, and completion. Record the resolved async flags and prove actual HIP combine and scatter execution.

Compare default, enabled, and rollback token streams on identical artifacts. Prove that removing the production dispatch makes the focused test fail. Historical measurements on PR #2779 showed no standalone throughput gain, so this issue owns functional ROCm parity rather than a performance claim.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the CUDA combine, scatter, and apply kernels and the shared runner, then inspect how HIP-only builds avoid CUDA headers. Add a production server or C API correctness case covering non-speculative admission, compaction or swap, and completion. Done means identical token streams across modes, resolved async flags, proven HIP execution, and a focused test that fails when production dispatch is removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.