[RFC]: Generalize `tebench` to Multi-Node Peer Benchmarking

### Changes proposed

## Summary

This RFC proposes extending `tebench` from a single-target benchmark tool into a general multi-node peer benchmark framework for Mooncake Transfer Engine and TENT.

The core idea is to model every `tebench` process as a **peer**. A peer may publish local memory, initiate transfers to remote peers, or do both. Topologies such as fan-out, incast, M-to-N, all-to-all, and hotspot traffic become different configurations of the same peer model.

This proposal does not introduce application-specific concepts such as provider/consumer. M-to-N is treated as one topology where M initiator peers access N target peers.

## Motivation

Current `tebench` is effective for simple single initiator to single target benchmarking, but it is not sufficient for validating production-like distributed transfer behavior.

Large-scale deployments rarely behave like a single isolated connection. Real workloads involve multiple peers establishing connections at the same time, accessing multiple remote segments, sharing or isolating remote address ranges, and creating topology-specific pressure on both the control plane and data plane.

Several important classes of issues are difficult or impossible to expose with single-target benchmarking:

- RDMA bootstrap storms when many peers connect concurrently.
- Control-plane scalability bottlenecks during simultaneous metadata and connection setup.
- Endpoint lifecycle bugs during peer restart, segment reopen, or teardown.
- Data-plane admission and completion accounting issues under all-to-all traffic.
- Traffic fairness problems when many initiators target the same peer.
- Data correctness risks when multiple initiators share or overlap remote ranges.
- Premature buffer release when one peer exits while other peers still access it.
- Performance cliffs that only appear under fan-out, incast, or hotspot traffic.

Without multi-node benchmarking support in `tebench`, these problems tend to be discovered late by downstream integration tests or production-like fault injection. Adding this capability upstream gives Mooncake a common, reproducible way to validate connection scalability, transfer correctness, and topology-level performance.

## Goals

1. Support multi-target transfers in the `tebench` binary.
2. Support multi-node orchestration through a reusable script.
3. Keep the model generic: peer, role, topology, target plan, range policy.
4. Cover common distributed traffic patterns:
- fan-out
- incast
- M-to-N
- all-to-all
- hotspot
- custom target lists
5. Preserve existing single-target behavior by default.
6. Preserve existing `tebench` capabilities such as TENT backend, classic backend, QoS, deadline-aware workloads, and mixed memory configurations.
7. Provide correctness workloads for shared remote range validation, especially `write_seed` and `read_verify`.

## Non-Goals

This RFC does not propose changing TENT transport protocols, RDMA connection management, metadata semantics, or runtime admission policies.

This RFC also does not define application-level ownership semantics. Terms such as provider and consumer are intentionally avoided. They can be represented by target and initiator peers when needed, but they are not part of the `tebench` model.

## Concepts

### Peer

A peer is one running `tebench` process.

A peer may:

- publish local memory as a target;
- initiate transfers to one or more remote peers;
- do both.

### Role

`tebench` supports three roles:

```text
target
initiator
peer
```

- `target`: publishes local memory and waits.
- `initiator`: opens remote target segments and transfers data.
- `peer`: publishes local memory and also transfers to remote peers.

### Topology

A topology describes how peers communicate.

| Topology | Description |
|---|---|
| `fanout` | One initiator peer accesses multiple target peers. |
| `incast` | Multiple initiator peers access one target peer. |
| `m2n` | M initiator peers access N target peers. |
| `all_to_all` | Every peer accesses every other peer. |
| `hotspot` | Multiple targets exist, but traffic is weighted toward one or more hot peers. |
| `custom` | Caller provides explicit target lists and weights. |

### Target Plan

Each initiator peer receives a target plan:

```text
target segment list
target weights
target offset
target range size
range policy
```

The target plan determines which remote peers are accessed and how traffic is distributed.

### Range Policy

Two range policies are proposed:

```text
shared
per_initiator
```

- `shared`: multiple initiators intentionally access the same remote range. This is useful for read-only validation, such as many readers verifying the same seeded data.
- `per_initiator`: each initiator gets an isolated remote range to avoid accidental data overwrite during write or mixed workloads.

## Proposed CLI Changes

The `tebench` binary should support:

```text
--mode=target|initiator|peer
--target_seg_name=
--target_weights=
--target_offset=
--target_range_size=
--op_type=read|write|mix|write_read_verify|read_verify|write_seed
--start_barrier_file=
--finish_ready_file=
--finish_release_file=
```

`--target_seg_name` remains compatible with the existing single-target format. A comma-separated list enables multi-target mode.

## Proposed Script

Add a reusable script:

```text
mooncake-transfer-engine/benchmark/scripts/run_tebench_multinode.sh
```

Suggested parameters:

```text
--nodes hostA,hostB,hostC
--targets hostA,hostB
--initiators hostC,hostD
--peers hostA,hostB,hostC
--topology fanout|incast|m2n|all_to_all|hotspot|custom
--target-instances N
--exclude-self-targets
--target-range-mode shared|per_initiator
--target-weights ...
```

The script is responsible only for orchestration:

- start target or peer processes;
- collect published segment names;
- build target plans;
- split remote ranges when needed;
- start initiators;
- coordinate finish barriers;
- clean up remote process groups;
- summarize logs and structured results.

Transport behavior remains inside `tebench` and TENT.

## Correctness Workloads

Two correctness-oriented operations are proposed:

### `write_seed`

Writes deterministic data to a remote range using a stable seed.

This prepares shared remote data for later read-only verification.

### `read_verify`

Reads remote data and verifies it against a stable seed without modifying the remote memory.

This enables multi-reader validation where many initiators read the same remote range concurrently.

These workloads are important because pure read/write performance counters are not enough. Multi-node benchmarking should also detect data mismatch, request loss, permanent pending states, and incomplete transfer accounting.

## Compatibility

Existing single-target usage remains valid:

```text
./tebench --target_seg_name= --backend=tent
```

Multi-target mode is enabled only when `--target_seg_name` contains multiple comma-separated segments.

If no multi-node script is used, `tebench` continues to operate as a standalone benchmark binary.

## Implementation Plan

The implementation can be delivered as one RFC-backed PR with layered commits:

1. Add multi-target CLI parsing and target list configuration.
2. Extend the benchmark runner interface to pass `target_idx`.
3. Extend TENT backend to open and manage multiple target segments.
4. Add weighted target selection and per-target statistics.
5. Add range and layout validation for multi-target workloads.
6. Add multi-node orchestration script.
7. Add `write_seed` and `read_verify` correctness workloads.
8. Update documentation and examples.

## Review Considerations

This change is expected to exceed 500 lines because it changes the benchmark model rather than adding a narrow test case. The code should therefore be reviewed with the RFC context in mind.

The main review focus should be:

- whether the peer/topology abstraction is generic enough;
- whether existing single-target behavior remains unchanged;
- whether multi-target changes preserve existing QoS, deadline, mixed-memory, and classic/TENT backend behavior;
- whether range isolation prevents accidental data overwrite;
- whether finish barriers prevent remote buffers from being released too early;
- whether the orchestration script remains transport-agnostic.

## Expected Benefits

This change gives Mooncake a first-class way to validate distributed transfer behavior.

It improves confidence in:

- RDMA connection scalability;
- control-plane behavior under concurrent bootstrap;
- endpoint lifecycle correctness;
- multi-peer data-plane stability;
- topology-specific bandwidth and latency;
- data correctness under shared and isolated remote ranges.

Most importantly, it moves critical multi-node validation upstream. Instead of each downstream project building its own integration harness, Mooncake can provide a common benchmark framework for reproducing, measuring, and preventing distributed transfer regressions.

### Before submitting a new issue...

- [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Start at the tebench binary and review the proposed CLI changes, then inspect the planned orchestration entry point at benchmark/scripts/run_tebench_multinode.sh. The implementation plan covers multi-target handling, topology orchestration, range policies, barriers, and correctness workloads; done means preserving single-target behavior while supporting the listed multi-node modes and validations.

Written by the indexing model from the issue text.