Enable EFA on the GPU node group for ModelExpress testing

Description

Prepare the test cluster so ModelExpress can use RDMA. EFA is the one fabric
scoped for validation.

An EFA interface can only be attached when an instance launches, so nodes without
one have to be replaced rather than reconfigured. Beyond the interface itself,
several node-level and network-level conditions have to hold before any traffic
moves, and none of them announce themselves when they are missing.

The changes land in the infrastructure repository that provisions the cluster.
This issue tracks the requirement and the outcome.

Definition of Done

• GPU nodes advertise the EFA resource to Kubernetes.
• Peer nodes share a subnet.
• The EFA device plugin is installed and tolerates the GPU node taint.
• The node group can reach the worker count the acceptance runs need.
• EFA is shown to carry traffic, not merely to be present.
• The change is captured somewhere reusable, not applied by hand only.

Resources

Five conditions had to hold. Each was found by hitting it:

1. interface_type = "efa" on the launch template network interface, plus node
   replacement, since the interface is attachable only at launch.
2. A single availability zone. EFA does not cross subnets.
3. The EFA device plugin, with a toleration for the GPU taint, otherwise the
   DaemonSet cannot schedule on the nodes that have the hardware.
4. The container runtime's locked-memory limit raised. RDMA registers pinned
   memory and the default is far too low, which fails ibv_reg_mr during engine
   init. This cannot be fixed from inside a pod.
5. An egress rule on the node security group referencing the group itself, for
   all protocols.

A sixth condition sat on this list for a while and turned out to be false:
that the host needs the AWS EFA software stack. It does not. On a stock Canonical
Ubuntu node with no /opt/amazon/efa and no efa-installer run, the in-tree
efa kernel module carries traffic at 5.26 GB/s. The DEREG_MR errors that
suggested a broken driver path were a symptom of item 5.

Item 5 was the blocking one and is worth stating plainly, because it is the least
obvious. Without it EFA is fully present and completely silent: the device shows
PORT_ACTIVE, ibv_devinfo is clean, the resource is advertised and schedulable,
and send counters increase while receive counters stay at zero. An allow-all
0.0.0.0/0 egress rule does not cover it, because EFA OS-bypass traffic is not
IP. This was isolated with a controlled test rather than inferred: removing the
rule broke EFA while TCP kept working, and restoring it recovered EFA.

A cluster placement group is a latency optimization, not a requirement. That was
worth confirming, since it is usually listed as mandatory.

Verification that the fabric actually works, rather than merely exists: run a TCP
control first, then ucx_perftest over the EFA transport, and require the EFA
hardware receive counters on both peers to increase. The counters are the part
that matters, because a fabric that silently falls back to another transport
still reports a successful transfer.

Part of #1113.

Start in the infrastructure repository at the GPU node group and its launch template, then trace the cluster networking and device-plugin configuration. Verify the node count, EFA resource, plugin scheduling, runtime limits, and security-group behavior. Run a TCP control followed by ucx_perftest and confirm EFA hardware receive counters increase on both peers.