bug: an RDMA member mounts the device tree but is granted no device

What happens

applyMemberFabric (pkg/worker/kvcache/mooncake/member_workload.go) gives a host-fabric member a
hostPath mount of /dev/infiniband plus IPC_LOCK and SYS_RESOURCE, and nothing else.

A hostPath mount of a device tree does not grant access to the devices inside it. Where the
kubelet drives the cgroup v2 device controller, a container's device allowlist is assembled from the
resources the Pod requests. A bind mount is not a request, so every character device under that path
is denied at open() no matter what its mode bits say.

How this was found, and what has been measured

MEASURED, on the EFA half of this same function: inside a running member container the EFA devices
were present and world-readable (crw-rw-rw-), the process ran as uid 0, and open() still returned
EPERM. Adding a device-plugin resource request to the Pod is what made them openable. That half is
fixed in #347.

MEASURED on the RDMA branch too. What this issue used to carry as an inference is now a reading,
taken on a node that enforces a device allowlist. Every absence below is paired with a reading that
must be present if the probe works, because "the device was not seen" is also what a broken probe
reports.

What the RDMA branch renders. The member Pod carries the device tree
(rdma-devices=/dev/infiniband) and requests {"requests":{"memory":"2Gi"}} — no device resource.
PAIRED: an EFA member rendered by the same operator on the same cluster carries
{"limits":{"vpc.amazonaws.com/efa":"1"},"requests":{"memory":"2Gi","vpc.amazonaws.com/efa":"1"}},
so the reading can see a device request when there is one, and the RDMA absence is not a blind read.

The device is present and cannot be opened. In a container on that node with the same mount and
the same lack of a device request, running as uid 0:

crw-rw-rw-. 1 root root 231, 192 uverbs0
/dev/infiniband/rdma_cm  REFUSED errno=1 Operation not permitted
/dev/infiniband/uverbs0  REFUSED errno=1 Operation not permitted
/dev/null                OPENED        <- paired: open() itself works

What the store makes of it, from the member's own log:

topology.cpp:67]  Device /dev/infiniband/uverbs0 is not accessible for read/write
topology.cpp:224] Skipping unavailable device: rdmap0s31
transfer_engine_impl.cpp:274] Topology discovery complete. Found 0 HCAs.
transfer_task.cpp:965] auto-detected: TCP-only environment

So it holds end to end: the mount arrives, the grant does not, open() is refused for uid 0 on a
world-readable device node, and the store reports no error while coming up on TCP.

Two limits on this reading, stated because they are the kind that get left out. The open()
lines were taken from a standalone Pod on the same node rather than from the member container: the
member was in CrashLoopBackOff and exec could not reach it. That Pod reproduces exactly the thing
under test — /dev/infiniband mounted, no device resource — and nothing else. The crash itself is
the test setup's fault and not a product defect: another host-network member was already bound to
port 8080 on that node, and the member exits with [Errno 98] address already in use. The transport
lines above are printed before that. The cluster has since been destroyed.

The exposure is environment-dependent, which is why it can have gone unnoticed: a node whose runtime
does not enforce a device allowlist will let the same rendering work. "Our RDMA members are fine" is
therefore consistent with this defect rather than evidence against it.

A second gap on the same branch

The EFA mount declares Type: Directory so that a node without the driver fails at mount time with a
message naming what is missing. The RDMA mount declares no type at all, so a node with no
/dev/infiniband starts the member anyway and fails later, quietly, inside the transport. The
comment on the EFA volume states exactly why the loud failure was chosen; the RDMA volume predates
that reasoning and never received it.

What deciding this looks like

The EFA fix names a resource the cloud's own device plugin advertises. RDMA has no single equivalent:
the resource name depends on which RDMA device plugin an administrator installed, and many clusters
install none. So the choice is real rather than mechanical:

• render a request for a configurable resource name, which adds API surface for something most
  clusters will leave unset;
• require a device plugin and refuse an RDMA backend without one, which is honest and narrows who can
  run the transport;
• or state that the operator stays out of it and document the node-level prerequisite, which keeps
  the current behaviour but stops it from being a surprise.

What would close this

An RDMA member, rendered by this operator on a node that enforces a device allowlist, opening an
/dev/infiniband device and completing a transfer over it.

What does NOT close this

• The EFA fix in #347. It settles one branch of a shared function and leaves the other as it is.
• A member Pod reaching Ready. Readiness does not open a device.
• An RDMA member working on a node that does not enforce a device allowlist, which is the
  configuration this defect is invisible in.
• A unit test over the rendered Pod, unless it asserts the grant rather than the mount. The mount is
  already there and is exactly what does not suffice.

References

• applyMemberFabric, pkg/worker/kvcache/mooncake/member_workload.go — the shared base, then the
  EFA-only remainder.
• #286 is a different gap on the same transport: engine Pods receive no fabric access at all. This
  one is about member Pods, where the access is rendered and does not take effect.

/kind bug
/area worker

Start in pkg/worker/kvcache/mooncake/member_workload.go at applyMemberFabric, comparing the RDMA and EFA volume and resource rendering. Decide how RDMA device access should be handled when resource names depend on the installed device plugin, and include coverage that verifies the device grant rather than only the mount. Done means an RDMA member can open /dev/infiniband and complete a transfer on a node enforcing a device allowlist.