[RFC]: Classic TE RDMA Two-Sided Transport (`rdma_twosided`)

### Changes proposed

## Summary

Add an **opt-in** classic Transfer Engine transport `rdma_twosided` for
PD-oriented workloads that prefer simpler buffer lifetime and TE-managed
bounce memory over peak one-sided bandwidth.

This is **not** a replacement for classic `rdma` (ibverbs WRITE/READ).
A process installs **either** `rdma` **or** `rdma_twosided`, not both.

Work is split into **four** reviewable PRs (large functional modules). Each PR
that introduces a runnable network path ships correctness tests plus a minimal
benchmark/smoke metric. End-to-end effectiveness vs one-sided is validated in
the final PR.

Related: #3324 (PR1, merged), #3440 (PR2, merged), #3613 (PR3, in review).

## Motivation

Classic one-sided RDMA requires applications to keep buffers registered for
the lifetime of inflight work and exposes remote rkeys to peers. For some PD
paths this is awkward:

1. **Buffer lifetime** — avoid repeated register/unregister around transfers.
2. **TE-owned networking buffers** — capacity can expand/shrink under credit.
3. **Decouple remote memory ownership** — SEND/RECV + local placement; no peer
app rkey for the default path.
4. **Acceptable latency** — bounce/copy cost is OK for the target PD scenarios;
peak GB/s remains on classic `rdma`.

## Non-goals

- Replacing classic `rdma` as the default high-bandwidth path.
- Implementing this behind libfabric for the first version (EFA/CXI already use
OFI for their fabrics; this path targets verbs/RoCE/eRDMA and stays on
ibverbs to align with classic `rdma`).
- Changing the existing TE handshake/discovery stack (only extend capability
fields as needed).

## Architecture

```text
TransferEngine / MultiTransport
├─ install("rdma") → classic one-sided (unchanged)
└─ install("rdma_twosided") → new transport (mutually exclusive)

rdma_twosided
├─ CtrlChannel (per-peer notify QP) — typed CtrlFrames, credit, notify
├─ MsgChannel (per-peer/rail msg QP) — SEND/RECV + bounce pool
└─ TE managed buffers + credit admission (WAITING when short)
```

## Roadmap (4 PRs)

### PR1 — Protocol primitives (#3324, merged)
**Deliverable:** `CtrlFrame` codec + `SenderCreditLedger` under
`transport/rdma_twosided/`, unit tests only.

**Validation:** unit tests (round-trip, epoch fencing, strict decode).
No network benchmark (no transport data path yet).

### PR2 — Control plane (#3440, merged)
**Deliverable:** `RdmaTwoSidedTransport` + `CtrlChannel` + handshake fields +
RDMA `sendNotify` (OOB fallback configurable).

**Validation:**
- Correctness: dual-engine notify tests (single / bidi / burst).
- Smoke metric: notify ops/s and latency on a real RDMA device.

### PR3 — Data-plane channel (#3613, in review)
**Deliverable:** `MsgChannel` + bounce expand/shrink, with credit used for
RQ/bounce admission.

**Validation:**
- Correctness: SEND/RECV loopback / dual-engine data messages.
- Smoke: bounce expand under pressure.

### PR4 — TE integration
**Deliverable:** managed-buffer APIs + `submitTransfer` two-sided path + docs;
wire PD install selection.

**Validation:**
- Correctness: managed WRITE/READ e2e, credit backpressure.
- Effectiveness: latency/bandwidth smoke vs classic one-sided; extend
tebench/validators where practical.

## Alternatives considered

| Option | Why not (for v1) |
|--------|------------------|
| Fold into classic `rdma` | Too invasive; hard to reason about and test. |
| libfabric / reuse EFA stack | Different fabric family; does not remove TE protocol work; weaker alignment with existing verbs `rdma`. |
| Single monolithic PR | Previously too large for review; split for reviewability and staged validation. |

## Open questions

- Final transport name (`rdma_twosided` vs something shorter).
- How aggressively tebench should grow in PR3 vs a dedicated follow-up.
- Whether credit-on-wire belongs entirely in PR2 or partially in PR3.

## Ask

Please review the motivation, the separate-transport boundary, and the 4-PR
validation plan. PR1 and PR2 are merged; PR3 is #3613 (in review); PR4 is
not yet opened.

Start with the four-PR roadmap and the linked PRs #3324, #3440, and #3613, then inspect transport/rdma_twosided/ and the tebench/validators areas mentioned in the issue. The planned work is complete only when PR4 adds managed-buffer and submitTransfer integration, install selection, documentation, and the listed correctness and effectiveness validation.

Written by the indexing model from the issue text.