[TENT] RDMA link-flap recovery gaps: hung WRs wedge QPs silently, swallowed bootstrap rejection, no PORT_ACTIVE fallback

# [TENT] RDMA link-flap recovery gaps: hung WRs wedge QPs silently (no error WC, no software timeout, no reconnect), plus swallowed bootstrap rejection and missing PORT_ACTIVE fallback

## Scenario

Prefill/Decode disaggregation (KV cache transfer) over RoCE with the TENT backend
(`MC_USE_TENT=1`), RDMA NICs are bonded (`mlx5_bond_*`).

**Observed behavior:** after PD transfer timeouts, or an RDMA NIC link down/up,
subsequent transfers never recover on their own. The classic (non-TENT) engine does
not show this behavior in the same environment.

## Controlled experiment (single-host repro)

`ip link set down`, wait ~4 min, `ip link set up`.

Timeline (all from logs):

- `08:18:06` — link down; all 8 engine processes correctly log `Action: mlx5_bond_4 down`.
- `08:19:03-08:19:54` — `fallback device selection` floods (~780k reroute attempts);
throughput degrades but does not drop to zero (other NICs still serve).
- `08:19:35+` — application-level 20 s timeouts (`batch_transfer_sync_write TIMEOUT`,
ret=-1).
- `08:21:54` — link up; all 8 processes correctly log `Action: mlx5_bond_4 up`
(so PORT_ACTIVE handling worked in this run).
- After link up: **partial recovery only**. All 4 pp_ranks submitted RDMA writes
(25 each), but pp_rank 1/2 succeeded 25/25 while **pp_rank 0/3 failed 25/25** —
and stayed broken until the end of the log (16+ min later).

The crucial part: for the failing ranks after link-up, the TENT layer logged
**absolutely nothing**:

- 0 × `Detected error WQE` (no error completions)
- 0 × `transfer timeout (software)` (slice-level software timeout never fired)
- 0 × `Unable to connect endpoint` (no reconnect attempts/failures)
- 0 × `Endpoint marked for destruction`

Slices were submitted at the app level and then vanished: never completed, never
errored, never reported by any TENT diagnostic path.

## Root cause analysis

### Defect A (primary, confirmed by experiment): hung WRs during link-down wedge the QP/endpoint forever

WRs posted to QPs on the downed NIC hang silently: while the port is down the
hardware neither transmits nor completes/errors them. After the link comes back,
those WRs **still never complete and never produce an error CQE**, so:

1. The QP's send-depth accounting (`wr_depth` / `cq_outstanding`) stays consumed
forever;
2. New slices posted to that endpoint either cannot be posted (QP full) or queue
behind dead WRs and are never sent;
3. Because these slices never reach the posted/completed/error paths, **none of
TENT's detection hooks fire**: no error WC (so no `disableEndpoint`), and the
software slice timeout never triggers for them, so the endpoint is never
destroyed and rebuilt;
4. The endpoint becomes a permanent blackhole. In our experiment the two ranks
whose paths crossed the downed NIC never recovered (16+ min, until shutdown),
while ranks on other NICs recovered immediately.

There is currently **no mechanism** in TENT that reclaims an endpoint with
outstanding WRs after a link recovery (e.g. destroying endpoints with stale
outstanding WRs on `PORT_ACTIVE`), and no visibility into stuck posted WRs
(the classic engine gained exactly this diagnostic in #2872 via
`MC_TRACK_RDMA_POSTED_SLICES`; TENT has nothing equivalent).

### Defect B: bootstrap RPC swallows the peer's rejection reason → misleading "Missing peer GID in bootstrap"

`ControlService::onBootstrapRdma` (`tent/src/runtime/control_plane.cpp`) ignores the
return value of `bootstrap_callback_`. When the peer rejects the bootstrap (stale
endpoint retired, device down, ...), the real reason goes into
`local_desc.reply_msg`, but the RPC still returns success with an empty `local_gid`,
and the initiator reports the misleading `Missing peer GID in bootstrap`
(`tent/src/transport/rdma/endpoint.cpp`). The classic engine fixed the equivalent
issue in #2959 (structured handshake rejection replies); TENT still has it at
v0.3.12.

### Defect C (latent, not triggered in this run): `PORT_ACTIVE` can be lost → context stuck in DEVICE_PAUSED forever

The async fd is registered with `EPOLLET` (`tent/src/transport/rdma/context.cpp`),
but `Workers::monitorThread` does `epoll_wait(..., maxevents=1, 100)` and
`handleContextEvents` reads exactly one `ibv_async_event` per wakeup
(`tent/src/transport/rdma/workers.cpp`). Under edge-triggered semantics, when
multiple events queue (e.g. a fast flap), `PORT_ACTIVE` may never be processed.
`resume()` has no other caller, and there is no periodic `ibv_query_port`
validation, so a context can stay `DEVICE_PAUSED` forever (making
`Workers::getEndpoint` fail all transfers on that local NIC permanently). The
classic engine is saved from this by its monitor worker unconditionally
re-activating contexts every second; TENT has no such fallback. In our controlled
experiment (down ~4 min, single events far apart) this race did not trigger, but
it remains a latent risk for fast flaps.

### Defect D: GID/LID cached at init and never refreshed

`RdmaContext` reads `gid_`/`lid_` once in `openDevice()` and always advertises these
cached values in bootstrap. If the GID changes after a flap while the process keeps
running, peers build AH entries from stale address vectors. The classic engine fixed
this in #2878 (refresh on HCA/GID change events); TENT still caches forever at
v0.3.12.

## Related classic-engine fixes (none of them touch TENT)

- #2872 — RDMA rail-failure handling + CQ timeout diagnostics (`MC_TRACK_RDMA_POSTED_SLICES`)
- #2959 — RDMA NIC failover recovery: local-RNIC handoff, `MC_RDMA_RAIL_PAUSE_SECONDS`,
structured handshake rejection replies
- #2878 — refresh RDMA metadata on HCA and GID change events
- #2941 — pause active reconnects to failed peers

Verified at tag v0.3.12: `tent/src/runtime/control_plane.cpp` and
`tent/src/transport/rdma/workers.cpp` still contain defects A–D.

## Suggested fixes for TENT

1. On `PORT_ACTIVE` (context resume), proactively destroy endpoints that still have
outstanding WRs instead of waiting forever for CQEs that will never arrive;
more generally, give an endpoint with stuck outstanding WRs a deadline after
which it is torn down and rebuilt.
2. Port the #2872-style posted-slice tracking / CQ-timeout diagnostics to TENT so
"slice posted but never completed" is visible.
3. Extend the software slice timeout to cover slices sitting in the per-worker
request queues waiting to be posted, not only posted inflight slices.
4. Propagate bootstrap failures through the control RPC (carry `reply_msg` or an
error status back to the initiator).
5. Drain the async event fd until `ibv_get_async_event` returns empty on each epoll
wakeup, and add a periodic `ibv_query_port`-based resume fallback.
6. Refresh the cached GID/LID on port/GID change events, like #2878 does for the
classic engine.

Happy to provide full logs or help test patches.

Run the documented single-host link-flap reproduction, then read tent/src/runtime/control_plane.cpp, tent/src/transport/rdma/context.cpp, tent/src/transport/rdma/workers.cpp, and tent/src/transport/rdma/endpoint.cpp. Trace endpoint recovery, async-event handling, bootstrap replies, and cached GID/LID use against the classic fixes cited in #2872, #2959, and #2878. Done means link flaps recover transfers, bootstrap rejection reasons reach the initiator, and fast PORT_ACTIVE events do not leave contexts paused.

Written by the indexing model from the issue text.