[Tracking] What has actually executed, on which host and which device: the evaluation matrix

Vx's claim is declare your machine, any machine. The evidence for it exists and is scattered
across commit messages, campaign walkthroughs, CI, and one directory added this week. Nobody can
currently answer "what has actually run, where" without reading all of it, and at least twice
recently a claim was made that the evidence does not support.

This issue is the inventory: hosts down the side, devices across the top, and for every cell either
a pointer to the artifact or a statement of what is missing and why.

What a cell has to mean

"It runs" is too weak to be worth tabulating, so cells are graded. The grades matter more than the
matrix does.

Tier	Meaning
T0	Compiles / is refused. No execution.
T1	Placement declared, type-checked, executed on the host shim. A non-CPU spawn on goes through runtime/host_dispatch.cpp (or the Apple backend), so the placement is enforced and the program runs, but nothing physically moves to a device.
T2	Placement declared, type-checked, executed on the placed device.
T2x	T2 or T1 across two machines of different architectures, in one run.

The distinction between T1 and T2 is the one that has caused trouble. A table that merges them reads
as "executes on four architectures" when what it means is "the same suite passes on two operating
systems". Any write-up must keep them apart.

The matrix

Host	no device	NVIDIA A100	NVIDIA H100	NVIDIA A10	NVIDIA B200	Apple ANE
aarch64 (Apple M4, macOS)	T1 ✅	—	—	—	—	EMPTY ❌
x86_64 (Linux)	T1 ✅	T2 ✅	T2 ✅	EMPTY	T0 only	n/a
aarch64 → x86_64	T2x ✅	—	—	—	—	—

Filled cells, with their artifacts

aarch64 / no device — T1. Full suite 183 passed, 0 failed
(docs/discussions/walkthrough_gpu_campaign_m0_2026_08_08.md §7). Of tests/backend/pass, 76 of 143
programs carry a // EXPECT: line, so they are compiled, JIT-linked, run, and their stdout compared
against a literal. 27 of those 76 contain transfer( or spawn on(, which is the placement-bearing
executed set. One test is macOS-gated (topology_dispatch.vx, // REQUIRES: macos).

x86_64 / no device — T1. The same 183/0 in M0 §7, and CI runs the suite on ubuntu-latest on
every push, so this cell is continuous rather than a one-off. Same 76 / 27 counts.

x86_64 / A100 — T2. Two sources.
utils/differential/ ran six paired programs on A100-SXM4-80GB, CUDA 12.8, -arch=sm_80, across two
drivers (570.172.08 and 580.159.04), 2026-09-02. Pair 06 additionally corroborated the machine model:
cudaGetDeviceProperties reported sharedMemPerMultiprocessor=167936 = 164 KiB, exactly what
fleet/a100-80.vx declares.
docs/discussions/walkthrough_gpu_campaign_m5_2026_08_10.md ran two A100-SXM4-80GB in one pod, NV12
between them, loopback deliberately so the network is out of the measurement.
utils/memalg/results/ holds 6 A100 measurement sessions from 2026-08-16.

x86_64 / H100 — T2. docs/discussions/walkthrough_gpu_campaign_m4_2026_08_10.md: prefill on one
H100 and decode on the other, 2x H100 80GB HBM3 with NV18. CUDA 12.8, driver 580.126.09, 64 tokens of
stories15M greedy. Tokens identical between VX_LLAMA_DISAGG=0 and =1, and identical to both
macOS CPU and Linux CPU. Peer transfers counted in bytes: peer 0 -> 1, 442368 bytes x2, 2775
device selections against 1174 peer copies.

aarch64 → x86_64 — T2x, and the strongest portability evidence in the repo. Commit 43357bb4
(2026-08-11), which added scripts/run_two_machine_demo.sh. An arm64 Apple laptop dispatching to an
x86-64 EC2 worker in us-west-2 over an SSH tunnel, ~37 ms round trip.

matmul local            -        -     0.349 s
matmul unplaced        64   384 KiB    3.080 s
kv local                -        -     0.281 s
kv unplaced            64   35.9 KiB   2.518 s
kv resident            50    7.1 KiB   1.967 s

Cross-architecture answers agree exactly (17.068802 and 113.135994), which establishes that the wire
format, the memref descriptors and the dispatch payload all survive a change of architecture --
nothing had checked that before. Placement does measurable work on a link with real latency: the KV
cache crosses once instead of eight times.

Two limits the commit states about itself, which any citation must carry: the worker is a CPU, so
this is a communication result and not an execution-performance one; and a fully placed matmul would
be 14 messages and 48 KiB but is not run, because with the result resident there is nothing to
read home and compare. That figure is a calculation, not a measurement.

The wall-clock seconds are one sample over one network on one day. The message counts, byte counts
and the two agreement values are deterministic and are what to quote.

Empty cells, and why

aarch64 / ANE — empty, and blocked by a filed bug rather than by hardware. Three independent
reasons, all in code:

1. The three ANE programs (ane_matmul.vx, ane_conv.vx, ane_attention.vx) carry no
   // EXPECT: line. Their only execution-looking directive is // EXEC_RUN:, and EXEC_RUN is
   implemented nowhere -- zero references in the test harness, zero in src/. Those lines are
   inert. The live RUN: lines check MLIR and LLVM text only.
2. The CoreML primitive models do not build: every cargo build prints Failed to compile matmul_4x4 with coremlc and Failed to compile affine_4 with coremlc. #366 is the cause --
   build.rs invokes a bare python3, so a venv coremltools is ignored. runtime/npu_dispatch.mm
   handles the missing model explicitly: Failed to load ANE model (matmul_4x4.mlmodelc)! Falling back to CPU.
3. Even with a model loaded, the dispatcher sets config.computeUnits = MLComputeUnitsAll, which
   asks CoreML to choose among CPU, GPU and Neural Engine. It does not pin execution to the ANE, so
   "ran on the ANE" needs external evidence (Instruments, powermetrics) that nothing records.

What does exist for Apple is the ANE dispatch path and its argument decoding, corrected in M1.
That is a code-correctness result and a much smaller claim. Until the three items above are
addressed, no ANE execution claim is supportable.

A10 — empty. No fleet model, no run, no artifact. The only A10 in the picture is an EC2 box that
was unreachable when checked.

B200 — T0 only. Appears in fleet/b200.vx and in utils/campaign/BASELINE.md's predicted
admission matrix, whose header records that it was generated 2026-08-03, before any hardware was
rented. There is no executed B200 run.

Note this cell has a design blocker as well as a hardware one: deviceChip() in
src/dialect/VxLowering.cpp defaults to sm_80 and is overridden by the VX_GPU_CHIP environment
variable, because a Topology declaration has no capability: field. A B200 run would therefore rest
on an environment variable rather than on the machine model, which is a sharp question to invite in a
project whose thesis is that the machine is declared.

What this is not

utils/campaign/BASELINE.md is a compile-time admission matrix -- 15 configurations x 6 SKUs, 90
cells, 67 admitted and 23 rejected -- and is regenerable by scripts/admission_matrix.sh. It answers
"which machines would accept this program", predicted before renting. It is a different and valuable
artifact, and it is not this one. Do not merge the two tables.

Work this implies

• Fix #366 so the CoreML models build, then give the ANE path an // EXPECT: test. That
  converts the ANE column from absent to T1 at least.
• Either implement EXEC_RUN or delete it from the three files that use it. An inert directive
  that looks like an execution check is worse than no directive.
• Decide whether an ANE execution claim needs CoreML-selection evidence, and if so, how it is
  captured reproducibly.
• A capability: field on the Topology declaration, so a device generation is declared rather
  than exported. Prerequisite for an honest B200 row.
• Keep this table current. It is the answer to "what has run", and it went stale by being spread
  across five artifacts.

Related: hiraditya/Vx.1#432 (the differential suite), hiraditya/Vx.1#366 (the ANE build), hiraditya/Vx.1#321 and hiraditya/Vx.1#348 (the two-machine run),
hiraditya/Vx.1#319 (the GPU campaign).

Start with the matrix in this issue and compare it against the cited artifacts, including the campaign walkthroughs, utils/differential/, utils/memalg/results/, scripts/run_two_machine_demo.sh, and utils/campaign/BASELINE.md. Inspect #366, runtime/npu_dispatch.mm, build.rs, and src/dialect/VxLowering.cpp for the blocked cells. Done means the inventory stays current, separates T1 from T2, and links each filled cell to evidence while explaining empty cells.