[RFC]: TENT RDMA Direct Path for Latency-Sensitive Transfers

### Changes proposed

## Summary

This RFC proposes a TENT RDMA fast path for latency-sensitive transfers. The
branch introduces a direct posting mode for requests that carry either:

* a non-zero `Request::deadline_ns`; or
* `IntentType::FOREGROUND_GET`.

When the direct path is available, the request bypasses the normal RDMA worker
queue and posts directly to a dedicated per-endpoint direct QP. Completion is
polled by the caller through `getTransferStatus()`. If the direct lane is busy,
the endpoint is not ready, or the request cannot be resolved to eligible RDMA
devices, the transport falls back to the existing worker-based path.

The goal is to improve burst latency for foreground traffic without disturbing
the high-throughput worker path used by normal transfers.

## Motivation

TENT's normal RDMA datapath is optimized for throughput:

1. requests are split into slices;
2. slices are assigned to worker queues;
3. workers select RDMA devices and endpoints;
4. workers batch/post slices to data QPs and poll shared CQs.

This is effective for large or steady transfer streams, but it adds queueing and
worker scheduling latency to short foreground transfers. In bursty mixed
workloads, foreground reads can wait behind already queued background slices even
when the foreground request itself is small and urgent.

The current branch shows better burst performance by reserving a small,
low-depth direct lane for latency-sensitive traffic. This lets foreground
requests avoid the worker queue when there is an idle direct lane, while still
preserving the existing worker path as the fallback and throughput engine.

## Problem Statement

The existing RDMA worker path couples two concerns:

* throughput-oriented batching and multi-slice scheduling; and
* foreground request progress.

Under foreground/background bursts, that coupling can hurt tail latency:

* the foreground request may wait for a worker wakeup;
* it may sit behind normal queued slices;
* it shares normal data QPs and CQ polling cadence;
* single-request latency is measured through a path optimized for aggregate
bandwidth.

We need a small, bounded fast path that can make progress immediately for
latency-sensitive requests, without turning all RDMA traffic into caller-posted
traffic.

## Goals

* Reduce burst latency for foreground/deadline-tagged RDMA transfers.
* Preserve the existing worker path for normal and fallback traffic.
* Keep direct-path concurrency bounded so foreground traffic cannot exhaust QP
or CQ resources.
* Support multi-rail direct transfer when a request spans enough bytes and
multiple local rank-0 RDMA devices are eligible.
* Keep endpoint lifecycle, bootstrap, cancellation, and completion accounting
compatible with the existing RDMA transport.
* Provide benchmark knobs that can reproduce foreground/background burst
scenarios and report instantaneous bandwidth.

## Proposed Design

### Direct-Path Selection

The RDMA transport treats a request as latency-sensitive when:

```cpp
request.deadline_ns != 0 ||
request.intent_type == IntentType::FOREGROUND_GET
```

For such requests, `RdmaTransport::submitTransferTasks()` first calls
`trySubmitDirect()`. If direct submission succeeds, no worker slices are queued.
If direct submission fails for any reason that indicates the fast path is not
available, the request continues through the normal worker path.

This makes the optimization opportunistic and backward compatible.

### Direct QP and CQ

Each `RdmaContext` owns a dedicated direct CQ:

```cpp
RdmaCQ* direct_cq_;
```

Each `RdmaEndPoint` owns a dedicated direct QP:

```cpp
ibv_qp* direct_qp_;
std::atomic direct_wr_depth_;
```

The direct QP is intentionally shallow:

* max send WR: 1;
* max recv WR: 1;
* one SGE;
* signaled sends.

This makes the direct lane a low-latency bypass, not a second bulk-transfer
engine. Normal data QPs remain responsible for sustained throughput and
queue-depth-heavy workloads.

### Direct Lane Ownership

Each `RdmaContext` has a single direct lane owner:

```cpp
std::atomic direct_lane_owner_;
```

`tryAcquireDirectLane(task)` succeeds only when the lane is idle. Completion
polling calls `releaseDirectLane(task)`. If a request is split across multiple
direct rails, the transport acquires all required context lanes before posting
any slice. If one acquisition fails, already acquired lanes are released and the
request falls back to the worker path.

This all-or-fallback behavior prevents partial direct submission before the
transport knows it has enough direct capacity.

### Device and Rail Selection

Direct submission resolves both local and remote memory through cached segment
metadata:

* local source buffer: `LOCAL_SEGMENT_ID`;
* remote target buffer: `request.target_id`;
* memory location: `bufferLocationForRange()`;
* eligible local NICs: rank-0 devices for the source memory entry filtered by
`rdma_batch->device_mask`;
* remote NIC: same device ID when the remote memory entry contains it, otherwise
round-robin across remote rank-0 devices.

For requests larger than `workers.block_size`, direct mode may split the request
across multiple local rank-0 devices:

```text
num_slices = min(eligible_source_devices,
ceil(request.length / workers.block_size))
```

The slice lengths divide the request range across the selected devices. Each
slice posts to the endpoint associated with its local context and remote device.

### Bootstrap Compatibility

The RDMA bootstrap descriptor carries the direct QP number:

```cpp
uint32_t direct_qp_num = 0;
```

Both active connect and passive accept include `direct_qp_num` in the bootstrap
exchange. If both peers advertise a direct QP, the endpoint configures that QP
with the peer direct QP number using the same RC QP state-machine helper used by
notification QPs.

Duplicate bootstrap detection includes `direct_qp_num` so an already established
endpoint is reused only when the direct QP generation also matches.

### Completion and Status

Worker threads do not poll the direct CQ. Direct completions are driven by
`RdmaTransport::getTransferStatus()`:

1. If the task is direct and still pending, poll the task's direct context.
2. If the task spans multiple direct contexts, poll all contexts.
3. For every WC, resolve the `RdmaSlice` from `wr_id`.
4. Complete endpoint direct quota.
5. Release the context direct lane.
6. Update slice/task status through the existing `updateSliceStatus()` helper.
7. On WC error, reset the endpoint.

Cancellation for a direct task marks `cancel_requested` and polls the direct CQ.
Already posted direct WRs are allowed to drain through normal completion, just
like already posted worker-path WRs.

### Worker Path Refactoring

The branch also moves endpoint lookup out of `Workers` into
`RdmaTransport::getEndpointForContextIndex()`. This lets both direct submission
and worker submission use the same endpoint construction and connection logic.

`bufferLocationForRange()` is factored into the RDMA buffer helper so direct and
worker routing choose memory-region-aware topology locations consistently.

### Benchmark Support

The benchmark changes support direct-path evaluation:

* `--tent_intent_type=` attaches an intent to sweep-mode TENT requests;
* `--workload_classes_json` carries per-class block size, batch size, deadline,
and intent;
* `--request_interval_us` adds aggregate pacing between issued transfers;
* measurement start is synchronized after warmup across benchmark threads;
* output includes `Avg Inst GB/s`, computed from per-transfer bytes and
transfer duration;
* average latency is derived from instantaneous bandwidth when available.

These changes make burst tests less sensitive to thread start skew and make it
easier to compare latency-sensitive direct mode against the worker path.

## Expected Behavior

### Foreground Burst

When a foreground burst arrives and direct lanes are idle:

* foreground/deadline requests post immediately on direct QPs;
* they avoid waiting behind background worker queues;
* caller-driven completion can observe progress without waiting for worker CQ
polling;
* larger foreground transfers may use multiple eligible direct rails.

### Direct Lane Busy

When the direct lane is busy:

* `trySubmitDirect()` fails fast;
* the request falls back to the normal worker path;
* correctness does not depend on direct mode availability.

### Normal Bulk Traffic

Requests without deadline and without `FOREGROUND_GET` continue using the
existing worker path. Bulk throughput should remain governed by worker queues,
normal data QPs, QP pools, and existing rail selection.

## Correctness and Compatibility

The direct path preserves the transport contract:

* a task is still represented as `RdmaTask`;
* every direct slice holds a task reference until completion;
* transferred bytes and final task status are updated by `updateSliceStatus()`;
* failed completions mark the task failed and retire the endpoint;
* endpoint destruction transitions direct QP to ERR and waits for direct WR
depth to drain, subject to the existing destruction timeout;
* if direct posting cannot be safely prepared, the normal worker path is used.

The bootstrap change is wire-compatible with peers that do not support direct
QP: `direct_qp_num == 0` means unsupported, and direct setup is skipped.

## Evaluation Plan

Run a foreground/background burst workload with and without foreground intent.
The foreground-intent run exercises the direct path; the unspecific-intent run
uses the normal worker path.

Target:

```bash
./tebench --backend=tent --xport_type=rdma --seg_type=DRAM
```

Baseline initiator:

```bash
./tebench \
--target_seg_name= \
--backend=tent \
--xport_type=rdma \
--duration=30 \
--start_num_threads=8 \
--max_num_threads=8 \
--request_interval_us= \
--workload_classes_json='[
{"name":"foreground","threads":2,"block_size":4096,"batch_size":1,
"intent_type":"unspec","slo_us":300,"weight":4},
{"name":"migration","threads":6,"block_size":4194304,"batch_size":2,
"intent_type":"migration","slo_us":0,"weight":1}
]' \
--qos_link_capacity_gbps= \
--qos_output_jsonl=rdma-worker-baseline.jsonl
```

Direct-path initiator:

```bash
./tebench \
--target_seg_name= \
--backend=tent \
--xport_type=rdma \
--duration=30 \
--start_num_threads=8 \
--max_num_threads=8 \
--request_interval_us= \
--workload_classes_json='[
{"name":"foreground","threads":2,"block_size":4096,"batch_size":1,
"intent_type":"foreground_get","slo_us":300,"weight":4},
{"name":"migration","threads":6,"block_size":4194304,"batch_size":2,
"intent_type":"migration","slo_us":0,"weight":1}
]' \
--qos_link_capacity_gbps= \
--qos_output_jsonl=rdma-direct-foreground.jsonl
```

Also run a deadline-based variant where the foreground class uses
`deadline_us` instead of `foreground_get`, because direct-path selection accepts
either signal.

### Metrics to Report

| Metric | Worker baseline | Direct fast path | Expected direction |
| ------ | --------------- | ---------------- | ------------------ |
| foreground `slo_attainment` | TBD | TBD | higher |
| foreground average transfer latency | TBD | TBD | lower |
| foreground `p99_us` | TBD | TBD | lower |
| foreground `p999_us` | TBD | TBD | lower |
| foreground `goodput_gbps` | TBD | TBD | higher |
| `weighted_goodput_gbps` | TBD | TBD | higher |
| aggregate throughput | TBD | TBD | similar or workload-dependent |
| background throughput | TBD | TBD | may decrease during foreground bursts |
| `Avg Inst GB/s` | TBD | TBD | diagnostic |

The RFC discussion should include the measured burst-performance data that
motivated this branch. The key claim should be foreground burst latency/SLO
improvement, not necessarily higher total link throughput.

## Risks and Mitigations

| Risk | Mitigation |
| ---- | ---------- |
| Direct path steals resources from bulk traffic | Direct QP depth and context lane ownership are capped at one outstanding direct task per context. |
| Foreground traffic exceeds direct capacity | Busy direct lane falls back to the worker path instead of blocking direct submission indefinitely. |
| Multi-rail direct partial submission creates inconsistent tasks | Acquire all required context lanes before allocating/posting direct slices; release all acquired lanes on failure. |
| Completion only progresses when status is queried | This matches latency-sensitive callers that actively wait on completion; benchmark should verify polling cadence effects. |
| Endpoint lifecycle misses direct WRs | Direct QP is included in ERR transition, outstanding-depth checks, destruction, and bootstrap generation matching. |
| Incorrect region-to-NIC mapping for coalesced buffers | Use shared `bufferLocationForRange()` in both direct and worker routing. |

## Alternatives Considered

### Tune Worker Priority Scheduling Only

Priority scheduling helps, but it still requires worker queueing and worker CQ
polling. The direct path removes that queueing component for eligible
foreground transfers.

### Add More Worker Lanes

More workers can reduce queueing in some cases, but they also increase resource
usage and do not guarantee that foreground requests avoid background batches.

### Use Normal Data QPs for Caller Posting

Posting directly to normal data QPs would couple foreground latency to existing
QP depth and worker queue accounting. A dedicated shallow direct QP keeps the
resource boundary explicit.

### Make Direct Mode Explicit in the API

The current branch avoids API churn by deriving direct eligibility from existing
deadline and intent fields. A future explicit policy knob could be considered if
applications need finer control.

## Open Questions

* Should direct mode have an explicit config switch, or is intent/deadline-based
automatic selection enough?
* Should direct completions also be polled by a lightweight background poller to
help callers that do not poll status frequently?
* Should the direct lane allow more than one outstanding WR on hardware where
shallow depth still preserves foreground latency?
* Should direct-path usage expose metrics such as attempted, succeeded, busy
fallback, post failure, and completion error counts?
* Should direct mode be limited to specific request sizes to avoid very large
foreground transfers occupying direct lanes too long?

## Acceptance Criteria

* Foreground/deadline-tagged requests use direct posting when all required
direct resources are available.
* Direct-path failure or resource contention falls back to the existing worker
path without changing request semantics.
* Direct QP numbers are exchanged and validated in bootstrap.
* Direct completions release endpoint quota, context lane ownership, and task
references exactly once.
* Endpoint destruction accounts for direct QP outstanding WRs.
* Multi-rail direct transfer can split one eligible foreground request across
multiple rank-0 RDMA contexts.
* `tebench` can compare worker baseline and direct foreground runs using
synchronized warmup, pacing, workload classes, intents, deadlines, and
instantaneous bandwidth reporting.

### Before submitting a new issue...

- [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Start with RdmaTransport::submitTransferTasks(), trySubmitDirect(), and getTransferStatus(), then trace the existing worker submission and completion paths they share. Run the tebench RDMA commands in the evaluation plan with and without foreground_get, and compare the listed latency, SLO, throughput, and Avg Inst GB/s metrics; done requires measured burst-performance data and the RFC’s compatibility and fallback behavior to be validated.

Written by the indexing model from the issue text.