NUMA: model the host's memory domains, then allocate into them
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 14
- Forks
- 2
- Avg merge
- 12h 42m
- Merged PRs (30d)
- 61
Description
NUMA is the one heterogeneous memory the fleet did not model, and the only one
that sits inside a machine we already own. This tracks taking it from "modelled"
to "exploited", in three stages. Stage 2 has landed.
Why this is the same problem, not a new one
A NUMA domain is a memory space that some execution units reach cheaply and
others reach across a link. That is structurally identical to two GPUs on a
node, which fleet/node-8gpu.vx already models as HBM + PEER_HBM with
priced NVLink edges between them. Nothing in the type system, the transfer cost
graph, or the capacity checker needed to change to say it.
What is different is the payoff. Every other space in fleet/ describes silicon
you have to rent. This one is in the build box the campaign already uses.
Stage 2 -- model it (LANDED, c3501613)
fleet/xeon-e5-2666v3.vx now declares its two domains rather than flattening
them. lscpu on the build box reports node0 = CPUs 0-8,18-26 and node1 =
9-17,27-35, so the 60 GiB is two 30 GiB domains with a real cost between them.
The file had predicted this itself, in a comment: "A tile larger than one node's
share is admitted as though it were local when part of it cannot be." It was
right, and that is now a refusal:
40 GiB tile, --machine fleet/xeon-e5-2666v3.vx
before: admitted
after: Error[E6009]: transferred tensor needs 42949672960 bytes
but memory space 'HBM' has capacity 32212254720 bytes
Both declared rates are derived rather than measured, and say so: 68.3 GB/s per
socket (4 DDR4 channels at 2133 MT/s) and 38.4 GB/s per direction across QPI
(two 1.1 links at 9.6 GT/s). Against each other they predict a remote access
costs 1.8x a local one.
Fixtures: tests/frontend/fail/numa_tile_larger_than_one_node.vx and
tests/optimizations/pass/numa_peer_node_is_priced.vx. The second asserts the
predicted picosecond costs, so it fails the day a measurement replaces a
derivation -- which is the point of stage 1.
Stage 1 -- measure it (READY TO RUN)
utils/campaign/run_numa_probe.sh measures all four (cpu node, memory node)
pairs on a two-socket box and compares the measured remote/local ratio against
the compiler's, reading the prediction out of --diagnostics-json so the script
cannot drift from the fleet file.
It compares ratios, not absolute times, and that is deliberate. The declared
rates are memory-controller peaks, and a copy moves two bytes of traffic per
byte copied, so the model over-predicts any single edge by at least 2x -- as
fleet/m4-uma.vx already records. Both sides of the ratio carry that error and
it cancels.
The script refuses on a machine with fewer than two NUMA nodes rather than
reporting something meaningless, so it is a no-op on an Apple Silicon laptop.
Outcome: within ~25%, replace the UNVERIFIED note on the QPI figure with the
measurement. Outside it, rule out the harness before the hardware -- buffer
larger than both LLCs, threads actually started on the bound node, box otherwise
idle.
Stage 3 -- exploit it (OPEN)
The compile-time half already reaches the runtime. A declared space gets a
stable dispatch id (MemorySpace::Custom(name) => fnv_dispatch_id(name)), and
the two nodes carry distinct ids into the emitted module -- 1855308478 for
HBM, 489426827 for PEER_HBM, pinned by the pass fixture.
vx_plugin_alloc_and_transfer in runtime/host_dispatch_common.h receives that
id and calls aligned_alloc, ignoring it. So a placement is checked and priced
at compile time and then lands wherever first-touch puts it.
What stage 3 needs:
- map the dispatch id to a NUMA node and allocate with
numa_alloc_onnode(or
mmap+mbind) - pin the dispatching thread to that node's CPUs
- an API wrinkle worth settling first:
vx_plugin_free(void*, uint32_t)
carries no size, andnuma_freerequires one. Either a size side-table or an
allocation header; the existing comment on that function is already firm that
an allocation made by a backend must be freed by the same backend.
Designing the stage 3 experiment honestly
Linux does first-touch plus AutoNUMA balancing, so the baseline is not
random placement and a comparison against worst-case placement would be a
strawman -- the same failure as a benchmark reporting a fourfold speedup for a
quarter of the work (#507).
The workload where first-touch actually loses is the common one: a buffer
allocated and touched by one thread, then consumed by threads on both sockets.
Report against three baselines, not one:
- naive first-touch (allocate in main, parallel-for later)
numactl --interleave=all- hand-tuned correct placement
The claim to support is "Vx gets you baseline 3 from a declaration, and refuses
the cases where you would have got it wrong" -- not "Vx is 2x faster than C".
Not in scope
The GPU SKU files are unaffected. A single GPU has one HBM domain and there is
nothing to split; the multi-device relation already lives in node-8gpu.vx.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with runtime/host_dispatch_common.h and the vx_plugin_alloc_and_transfer/vx_plugin_free entry points, then inspect the dispatch IDs and fleet/xeon-e5-2666v3.vx. Run utils/campaign/run_numa_probe.sh and review the named NUMA fixtures to understand the current model. Stage 3 is done when dispatch IDs select NUMA placement, freeing remains correct, and the three-baseline experiment reports results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux
- Domain
- operating-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100