mudler / mudler/vllm.cpp

ROCm's device-weight budget is hipMemGetInfo's 64.00 GiB total, but every allocation on an integrated board is managed and tops out at a measured 58.000 GiB

Open
#2,518 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-ROCM-LANE-GUARD

RocmPlatform::residency_policy().device_memory_total_bytes is probed with
hipMemGetInfo (vt::rocm::DeviceMemoryTotalBytes,
src/vt/rocm/rocm_backend.hip:491-499), and that is what
DeviceWeightBudgetBytes hands CheckDeviceWeightFit as the pool the load is
measured against. On strix:gpu0 (gfx1151) it reports 68719476736 B
(64.00 GiB)
.

But on this board UseManagedAlloc(caps) is true — integrated=1 managedMemory=1 concurrentManagedAccess=1 — so every Backend::Alloc is
hipMallocManaged (src/vt/rocm/rocm_backend.hip:163-181). The measured
ceiling of that allocator, three consecutive samples in one probe run
(/mnt/nas_share/rc/glm53-rocm/out2/probe2.log:58-65):

hipMallocManaged: stop at 58.000 GiB (out of memory)
hipMallocManaged CEILING = 58.000 GiB (62277025792 bytes)
hipMallocManaged CEILING = 58.000 GiB (62277025792 bytes)
hipMallocManaged CEILING = 58.000 GiB (62277025792 bytes)

68719476736 - 62277025792 = 6442450944 B = 6.00 GiB of budget the allocator
cannot reach
, and the error is in the optimistic direction: a load sized
between the two figures passes the #1123 refusal and then dies in
ResidentWeight::Alloc, which is the late OOM that refusal exists to prevent.

The board's own system RAM is 62 GiB total / 58 GiB available
(.agents/specs/rocm-glm53-dsa.md), so the 58.000 GiB ceiling is the host memory
the managed allocator actually has, not a driver quirk — hipMemGetInfo's
"total" on an integrated part describes an aperture, not an allocatable pool.

Why it is filed rather than fixed under BACKEND-ROCM-LANE-GUARD

That row moves one predicate in the streamed-expert lane guard. Changing what
device_memory_total_bytes MEANS on ROCm moves the refusal for every ROCm load
on every model, which is exactly the scope #1934 deliberately left at the raw
probe. It wants its own row and its own before/after hardware evidence.

What a fix owes

A ceiling that reflects the allocator actually in use — bounded by
UseManagedAlloc, since a non-managed ROCm board's hipMalloc pool is a
different number — without turning a once-probed constant into a repeated
allocate-until-failure sweep at every process start. Note the probe above takes
real time and real memory pressure, so a naive "measure it at registration"
implementation is not free.

Relevant on this board today because the hybrid placement's own reduced figure
(68634321408 B, see #2517) sits ABOVE the 62277025792 B ceiling while sitting
below the 68719476736 B budget, so the two issues interact: honouring the plan
without correcting the budget would move the failure rather than remove it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with RocmPlatform::residency_policy(), DeviceMemoryTotalBytes in src/vt/rocm/rocm_backend.hip:491-499, and the managed-allocation path at lines 163-181. Review the hardware evidence in probe2.log and .agents/specs/rocm-glm53-dsa.md, then compare the budget with the measured allocator ceiling. Done means the ceiling reflects the allocator selected by UseManagedAlloc without a repeated allocate-until-failure startup sweep, with before/after evidence on the affected hardware.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.