todo: the EFA functional pass is in and the headline numbers are not

What is left undone

Two things, after a cluster run took all four criteria.

1. The headline numbers were not taken. This issue offered two tiers, and the run took the
   cheaper one: a functional pass on a 50 Gbps EFA instance. Read and write came in at 3.62 GB/s and
   3.68 GB/s, which is 59 percent of line rate — so the ceiling measured here is the NIC, and
   criterion 3 asks for a path whose ceiling is DRAM. Whether to spend a p5-class instance for that
   number is a cost decision, not a technical one.
2. status.members[].protocol still is not an observation. Criterion 1 asks that the field
   report what the leader observed. It reported efa and the member genuinely was on efa, so the
   value was right — but it was right because the declared value happened to match, not because
   anything observed it. That mechanism is #341.

What has been measured

On two EFA-capable instances, against gpustack/mirrored-mooncake:0.3.13.post1-cpu built from the
commit that merged #347, with the operator built from a branch carrying the same fix.

Criterion 1 — the member comes up on efa. Confirmed on a member this operator rendered, not a
hand-built stand-in. Its startup log reports Found 1 HCAs, the EFA device line with
provider: efa, installTransport, type=efa, and a 4 GiB segment mounted. The leader reports
MembersMounted=True, state=OK.

Criterion 2 — a transfer crosses the fabric. A single member pinned to one node with the client
on its peer, so the store held exactly one segment and it was not on the client's side: every byte
had to cross. This removes the local-copy path rather than arguing it was not taken. For a 512 MiB
write plus a 512 MiB read, both ends showed matched bidirectional deltas of ~538.85 MB against a
536,870,912-byte payload, the difference being protocol overhead. An idle 30-second control gave
deltas of 0.

rdma_write_bytes / rdma_read_bytes read 0 throughout. The EFA provider uses SRD send/recv,
counted in tx_bytes/rx_bytes. That 0 is an inapplicable counter, not an absence of traffic.

Criterion 3 — CPU-to-CPU. --protocol=efa, 1 MiB blocks, batch 128, 12 threads. Functional pass;
see above for why the number is not the headline one. Two deviations from the criterion's wording are
worth recording:

• This build carries no --use_vram flag. CPU is the default and the target announces
  DRAM is used, numa node num: 1.
• ⛔ --auto_discovery=true overrides --protocol=efa. Given both, the engine tries to install
  rdma, fails on Failed to create completion queue: Operation not supported [95], and then
  produces numbers anyway. Anyone not reading the transport line gets a reading from a path they
  did not select.

Criterion 4 — a node without the fabric stops the member. Confirmed, and the mechanism changed;
see the next section.

Two parts of the criteria are now worded against behaviour #347 replaced

Both were accurate when filed and are not any more. Recorded here rather than silently rewritten.

The /opt/amazon/efa mount is gone. Criterion 1 expects the member to mount it and criterion 4
names the Directory type check as the thing that stops an unequipped node. #347 removed the mount:
the image now carries the matched libfabric, and the member requests the EFA device resource instead.

So criterion 4's refusal moved from the kubelet to the scheduler, and both shapes were captured:

	old	new
who refuses	kubelet, after scheduling	scheduler, before it
message	MountVolume.SetUp failed ... /opt/amazon/efa is not a directory	0/4 nodes are available: 1 Insufficient vpc.amazonaws.com/efa, 3 node(s) didn't satisfy plugin(s) [NodeAffinity]
Pod	Pending	Pending
condition	MembersMounted=False	MembersMounted=False, reason Unschedulable
status.members	empty	empty

The new shape is not the quieter one, which is what criterion 4 exists to protect. It names the
missing resource in the condition message, where the old one named a path and left the reader to work
out why that path mattered. And the old shape had a failure mode the new one does not: a node that
has /opt/amazon/efa but no usable device passed the mount check, started the member, got no
device-cgroup grant, and fell back to TCP silently — while status.members[].protocol still read
efa. That silent pass is what the device request removes.

What to run for what remains

On a p5 / p5e / p5en / p6 instance pair, transfer_engine_bench --protocol=efa with 1 MiB blocks,
reporting read and write throughput.

The cluster this round used cannot answer it, and that is a property of the instance family rather
than of the run. It is two c5n.9xlarge EFA nodes and two g6.xlarge; none of them is p5-class.
Recorded so the next reader does not take this cluster shape for one that could take the headline
half, and so that keeping such a cluster alive is not mistaken for keeping this issue's remaining
work within reach. ⛔ Do not pass --auto_discovery=true alongside --protocol,
and read the transport line before trusting any number.

What to watch for

• Instance family matters: i7ie is not EFA-capable (EfaSupported: false, measured via
  describe-instance-types) — the storage-optimized family the KV-cache e2e clusters used cannot run
  this. Pick the instance first.
• A stock node in a managed Kubernetes offering may have no /opt/amazon/efa at all: the AMI can
  run the EFA installer in its minimal mode, which omits libfabric. Since #347 the image carries its
  own, so this no longer blocks a member — but it does mean a host-side fi_info may be absent on a
  node whose members are working.
• A hostPath mount of a device tree does not grant access to the devices in it where the runtime
  enforces a device allowlist. The EFA path now requests the device resource; the RDMA path does not,
  which is #348.
• Same-host loopback on DRAM buffers is safe (SHM memcpy path); the documented segfault
  (ofiwg/libfabric#12328) applies to GPU buffers only, which this scope does not mount.

What does NOT close this

• The functional pass above. It is recorded as a functional pass because the criterion offered
  that tier explicitly; it is not the headline number and does not become one by being the only one
  taken.
• A number from a run that also passed --auto_discovery=true. That run installs a different
  transport, fails, and still prints throughput.
• status.members[].protocol reading efa. It reads the declared value; agreement with reality
  is not evidence of observation. See #341.

References

• Upstream design doc: docs/source/design/transfer-engine/efa_transport.md in kvcache-ai/Mooncake
  (CPU-to-CPU benchmark tables included).
• Engine Pods still get no fabric access on any host-fabric path: #286.

Start with docs/source/design/transfer-engine/efa_transport.md, then select a p5, p5e, p5en, or p6 instance pair and run transfer_engine_bench --protocol=efa with 1 MiB blocks. Do not combine --auto_discovery=true; verify the transport line before trusting results. Done means reporting read and write throughput for the CPU-to-CPU headline benchmark.