Port async device-mirror combine/scatter kernels to ROCm
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-ROCM
The async device-mirror path eliminates the D2H sampled-token round trip by keeping last_sampled_tokens device-resident and splicing them into input_ids on the GPU. The current ROCm backend does not advertise this capability, so the shared runner does not reach the ROCm combine and scatter path.
Port the CUDA combine, scatter, and apply kernels to HIP in a current-main change. Route them through the shared runner without pulling CUDA headers into HIP-only builds. Add a production server or C API correctness case with non-speculative admission, compaction or swap, and completion. Record the resolved async flags and prove actual HIP combine and scatter execution.
Compare default, enabled, and rollback token streams on identical artifacts. Prove that removing the production dispatch makes the focused test fail. Historical measurements on PR #2779 showed no standalone throughput gain, so this issue owns functional ROCm parity rather than a performance claim.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the CUDA combine, scatter, and apply kernels and the shared runner, then inspect how HIP-only builds avoid CUDA headers. Add a production server or C API correctness case covering non-speculative admission, compaction or swap, and completion. Done means identical token streams across modes, resolved async flags, proven HIP execution, and a focused test that fails when production dispatch is removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100