mudler / mudler/vllm.cpp

Build the pinned vLLM oracle for ROCm/gfx1200 in Docker, with a benchmarking/profiling harness against vllm.cpp

Open
#2,961 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: ORACLE-VLLM-ROCM-GFX1200-DOCKER

What this is

A standing Docker project that builds the pinned PRIMARY oracle
(e126687a9a, vLLM 0.28.1rc1.dev132) for ROCm on gfx1200, alongside
vllm-gguf-plugin, and runs benchmarking/profiling comparisons against
vllm.cpp's own HIP build on the same box (RX 9060 XT).

This mirrors the measurement BACKEND-GATE-ROCM-LLAMACPP already did by hand
on gfx1151 (#2740,
docs/bench-evidence/oracle-vllm-gfx1151-20260903.md), but packages the
recipe as reusable infrastructure instead of a one-off rc job, and targets
gfx1200 instead of gfx1151.

Why

  • .agents/oracles/vllm.md's device-scoped gateability table is empty at the
    CURRENT pin (e126687a9a) on every device. Nobody has built or run the
    primary oracle on ROCm at this pin yet.
  • gfx1200 is confirmed in HIP_SUPPORTED_ARCHS and the default
    PYTORCH_ROCM_ARCH at this exact commit (checked directly against
    CMakeLists.txt:52 and docker/Dockerfile.rocm_base:109 at
    e126687a9a), so the prerequisite that blocked earlier attempts on other
    devices does not obviously apply here — unmeasured, not assumed.
  • This box's own GPU (RX 9060 XT, gfx1200) is the only local hardware
    available, so a CUDA oracle image is fleet-only; a ROCm image is what
    makes apples-to-apples comparison possible without a lease.

Scope

  • Dockerfile building the pinned vLLM commit for
    VLLM_TARGET_DEVICE=rocm PYTORCH_ROCM_ARCH=gfx1200, plus
    vllm-gguf-plugin at its recorded pin, plus the five packages
    oracle-vllm-gfx1151-20260903.md found missing from a bare ROCm image
    (python3-dev, rocm-libs, libdrm-dev, ROCm torchvision, amdsmi).
  • A platform-identity smoke test baked into the image (mirrors
    gen_rocm.py's asserts): HSA_OVERRIDE_GFX_VERSION unset,
    RocmPlatform resolved, not UnspecifiedPlatform.
  • A benchmarking/profiling script comparing the containerized vLLM oracle
    against a host-built vllm.cpp binary: same prompts, same token counts,
    same GPU (device passthrough), GPU-clock-state discipline per
    .agents/benchmarking.md.

Out of scope

  • Advancing the vLLM parity pin or any src//include/ change.
  • A performance verdict of any kind before the declared token gate passes
    (per AGENTS.md §Gates); this row builds the tool, it does not itself
    license a benchmark claim.
  • CUDA oracle packaging (fleet-only, separate row if wanted).

Blocker to record

This session's user does not have docker group membership on the box the
work is being done on, so the image could not be built or run to verify
inside this session. sudo usermod -aG docker <user> plus a new login
session unblocks it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .agents/oracles/vllm.md, AGENTS.md §Gates, .agents/benchmarking.md, and docs/bench-evidence/oracle-vllm-gfx1151-20260903.md, then inspect the pinned vLLM commit and existing Dockerfile conventions. The work is done when a gfx1200 ROCm image builds with the specified pins and platform smoke test, and its harness compares it with the host vllm.cpp build under the declared benchmarking discipline.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, docker, python
Domain
ai, devops, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.