[Feature Request]: Auto-refresh segment metadata on IBV_WC_REM_ACCESS_ERR (flag-gated), keeping metacache fast AND correct

## Summary

Add an opt-in flag (e.g. `MC_SYNC_ON_REM_ACCESS_ERR=1`) that makes the RDMA transport automatically call `TransferMetadata::syncSegmentCache()` when a work completion fails with `IBV_WC_REM_ACCESS_ERR`, then re-post the failed slices once with the refreshed descriptor.

This keeps the full performance benefit of `metacache=true` (zero metadata RPCs in steady state) while restoring correctness through *error-driven* invalidation — and it fixes the problem **inside** Transfer Engine, so upper layers (vLLM / SGLang / custom EPD stacks) need no per-caller adaptation.

## Problem

With `metacache=true` (the default), a segment descriptor is fetched once per peer and never refreshed. There is no TTL, no invalidation channel, and `deleteEndpoint`/QP rebuild does **not** touch the metadata cache. So whenever the receiver's registration state changes while the sender holds a cached descriptor, every write to the affected range fails deterministically:

```
receiver dereg/re-register (or process restart / OOM-kill & respawn)
→ sender still resolves the target address against the frozen snapshot
→ posts WRITE with a stale rkey
→ IBV_WC_REM_ACCESS_ERR
→ worker_pool tears down the endpoint and rebuilds QPs
→ rebuilt QP posts the SAME stale rkey (cache untouched)
→ error → teardown → rebuild → ... (self-sustaining storm)
```

The recovery action (endpoint rebuild) repairs the wrong layer: the fault is metadata staleness, not connection health. Every sender engine that cached the descriptor runs this loop concurrently, which multiplies QP churn (adjacent effect discussed in #3299).

## Evidence

We hit this in production multimodal disaggregated serving (encoder→prefill embedding writes over Transfer Engine, P2P handshake mode) and verified the mechanism at the verbs level on **two silicon families**:

* A 4-phase reproducer (register → handout → dereg + same-VA re-register → stale write) produces `IBV_WC_REM_ACCESS_ERR` with zero bytes landing, and kills the work QP (identically on NVIDIA mlx5 / RoCE v2).
* On both vendors the rkey turned out to be a deterministic function of (VA, length) — same buffer re-registered gets the same rkey — which explains why the failure is *intermittent* in production: only the dereg→re-register window and layout-shifted ranges misfire, making the storm hard to attribute.
* Production A/B over the same 1-hour window, same cluster, same workload family: fleet with `MC_DISABLE_METACACHE=1` served 11.5M log-lines of traffic with **0** remote-access errors; the control fleet (metacache on) produced 430+ `Process failed for slice ... remote access error` in the same window.

## Why existing mitigations are not enough

| Option | Limitation |
|---|---|
| `MC_DISABLE_METACACHE=1` | Correct, but pays one descriptor RPC per transfer batch forever; discards the cache design entirely |
| Periodic refresh polling (#2795, `MC_TE_METADATA_REFRESH_INTERVAL_SECONDS`) | Bounded staleness, not zero: every failure inside the polling window still surfaces to callers; background cost scales with peers × frequency |
| `syncSegmentCache()` API | Exists since early releases, but nothing calls it on the failure path, and it is not exposed through the Python bindings — every integrator has to rediscover this problem and build their own hook |

Error-driven sync composes with (and is strictly cheaper than) both: no steady-state cost, and the staleness window collapses from `interval` to one failed WR.

## Proposed design

1. **Flag**: `MC_SYNC_ON_REM_ACCESS_ERR=1` (default off for compatibility).
2. **Trigger**: in the worker-pool completion path, when status is `IBV_WC_REM_ACCESS_ERR` (arguably also `IBV_WC_REM_INV_REQ_ERR`), resolve the peer segment of the failed slice and call `metadata_->syncSegmentCache(segment_name)` *before* the slice is retried / the task is failed.
3. **Throttle**: per-segment cooldown (e.g. 1–5 s) so a burst of failures against one peer triggers at most one refresh; concurrent failures wait on the in-flight sync rather than stacking RPCs.
4. **Scope**: refresh only the affected segment, not the whole cache.
5. **Retry**: after a successful refresh, re-resolve the descriptor and re-post the failed slices once (bounded), so a metadata-only fault never surfaces to the caller.
6. **Interaction with endpoint teardown**: refresh should complete (or at least be initiated) before the rebuilt endpoint re-posts, otherwise the new QP replays the stale key — this ordering is the crux of the storm today.

## Relationship to existing work

RFC #2762 ("Reliable Metadata Across Node Replacement") already lists *"On transport failure, force a refresh instead of waiting for TTL"* as step 5 of its recovery protocol. This FR is a narrowly-scoped, independently shippable first increment of exactly that step, gated behind a flag, with no wire-format changes (instance-id / snapshot semantics from the RFC can land later on top).

We are happy to contribute the implementation PR (and a unit/integration test based on our dual-vendor reproducer) if maintainers agree with the direction.