ROCm's device-weight budget is hipMemGetInfo's 64.00 GiB total, but every allocation on an integrated board is managed and tops out at a measured 58.000 GiB
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-ROCM-LANE-GUARD
RocmPlatform::residency_policy().device_memory_total_bytes is probed with
hipMemGetInfo (vt::rocm::DeviceMemoryTotalBytes,
src/vt/rocm/rocm_backend.hip:491-499), and that is what
DeviceWeightBudgetBytes hands CheckDeviceWeightFit as the pool the load is
measured against. On strix:gpu0 (gfx1151) it reports 68719476736 B
(64.00 GiB).
But on this board UseManagedAlloc(caps) is true — integrated=1 managedMemory=1 concurrentManagedAccess=1 — so every Backend::Alloc is
hipMallocManaged (src/vt/rocm/rocm_backend.hip:163-181). The measured
ceiling of that allocator, three consecutive samples in one probe run
(/mnt/nas_share/rc/glm53-rocm/out2/probe2.log:58-65):
hipMallocManaged: stop at 58.000 GiB (out of memory)
hipMallocManaged CEILING = 58.000 GiB (62277025792 bytes)
hipMallocManaged CEILING = 58.000 GiB (62277025792 bytes)
hipMallocManaged CEILING = 58.000 GiB (62277025792 bytes)
68719476736 - 62277025792 = 6442450944 B = 6.00 GiB of budget the allocator
cannot reach, and the error is in the optimistic direction: a load sized
between the two figures passes the #1123 refusal and then dies in
ResidentWeight::Alloc, which is the late OOM that refusal exists to prevent.
The board's own system RAM is 62 GiB total / 58 GiB available
(.agents/specs/rocm-glm53-dsa.md), so the 58.000 GiB ceiling is the host memory
the managed allocator actually has, not a driver quirk — hipMemGetInfo's
"total" on an integrated part describes an aperture, not an allocatable pool.
Why it is filed rather than fixed under BACKEND-ROCM-LANE-GUARD
That row moves one predicate in the streamed-expert lane guard. Changing what
device_memory_total_bytes MEANS on ROCm moves the refusal for every ROCm load
on every model, which is exactly the scope #1934 deliberately left at the raw
probe. It wants its own row and its own before/after hardware evidence.
What a fix owes
A ceiling that reflects the allocator actually in use — bounded by
UseManagedAlloc, since a non-managed ROCm board's hipMalloc pool is a
different number — without turning a once-probed constant into a repeated
allocate-until-failure sweep at every process start. Note the probe above takes
real time and real memory pressure, so a naive "measure it at registration"
implementation is not free.
Relevant on this board today because the hybrid placement's own reduced figure
(68634321408 B, see #2517) sits ABOVE the 62277025792 B ceiling while sitting
below the 68719476736 B budget, so the two issues interact: honouring the plan
without correcting the budget would move the failure rather than remove it.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with RocmPlatform::residency_policy(), DeviceMemoryTotalBytes in src/vt/rocm/rocm_backend.hip:491-499, and the managed-allocation path at lines 163-181. Review the hardware evidence in probe2.log and .agents/specs/rocm-glm53-dsa.md, then compare the budget with the measured allocator ceiling. Done means the ceiling reflects the allocator selected by UseManagedAlloc without a repeated allocate-until-failure startup sweep, with before/after evidence on the affected hardware.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100