Run ModelExpress scale-out, fallback, and cleanup acceptance

Description

Validate the documented configuration end to end on a real GPU cluster, and
produce the evidence the guide's claims rest on.

A healthy workload proves nothing on its own. A worker that cannot use
ModelExpress falls back and still serves, and a fabric mismatch still transfers
over a slower path. Both failure modes are silent, so the runs have to check the
transport rather than the health of the workload.

Definition of Done

• Worker counts of 2, 4, and 10 are each run, so consistency is visible rather
  than assumed.
• Each ModelExpress run has a matched native-loader baseline with identical
  model, image, node type, cache state, and parallel shape.
• Evidence shows the intended NIXL backend was used. A transfer over a fallback
  path is not a validated accelerated configuration.
• Evidence shows later workers resolved a peer source rather than the upstream.
• Fallback is forced, and the worker reaches Ready on the fallback path rather
  than failing. Serving correctness on that path was not separately exercised, so
  the evidence covers the load path and readiness, not a response body.
• Scaling the workers back to zero releases the GPU and EFA resources, so a
  repeat run starts from a clean scheduling state.
• Raw intervals and conditions are reported, not a single summary percentage.

Full teardown of the release, server, CRDs, and namespace is documented in the
guide but was not exercised, since the cluster stayed up across the whole
measurement sequence. Worth running once against a throwaway namespace.

Resources

Runs were done on ten g4dn.8xlarge nodes in one availability zone, one GPU and
one EFA interface each, with Qwen/Qwen3-0.6B.

Transport was confirmed at the hardware level rather than from application logs.
EFA receive counters on the receiving node were read before and after each run,
so a transfer that had quietly fallen back to another path would not have counted
as a pass.

The headline result is that the answer changes with worker count. ModelExpress is
neutral at 2 and 4 workers and materially slower at 10. The cause is that a single
source serves a fixed aggregate bandwidth and divides it among readers, so
transfer time grows roughly linearly with the number of simultaneous receivers.
Meanwhile the baseline barely degrades as workers are added, because a CDN-backed
hub absorbs fan-out.

One methodological note that cost real time and is worth passing on. The
Model loading took log line is not comparable between the two arms: it includes
the peer transfer when ModelExpress is on, but excludes the hub download when it
is off, because that fetch happens earlier in startup. Comparing them produced a
result that was wrong in direction, not just in size. Wall-clock time from
container start to pod Ready counts the same work on both sides and is the
metric these results use.

Each result is a single run under the stated conditions. The 10 worker pair is the
most reliable, since both arms ran on the same ten nodes; the 4 worker arms shared
only two nodes, so placement is confounded with the condition there.

Part of #1113.

Start with the guide's documented ModelExpress configuration and the release, server, CRDs, and namespace entry points, then run the stated sequence on the ten-node GPU cluster. Compare matched ModelExpress and native-loader runs at 2, 4, and 10 workers, force fallback, inspect EFA counters and pod Ready timing, and scale to zero. Done means raw intervals and conditions document transport, peer resolution, fallback readiness, and GPU/EFA cleanup, with one throwaway-namespace teardown run.