[RFC][Store]: NUMA-aware DRAM allocation domains and placement

### Changes proposed

## Summary

This RFC proposes NUMA-aware DRAM allocation domains and placement for Mooncake Store.

Mooncake already discovers NIC NUMA nodes in TENT, can bind different regions of a Store buffer with `mbind()`, and registers the resulting location map with Transfer Engine. However, this locality is not represented in the Store Master's placement model: a mounted `Segment` currently carries host identity but no NUMA identity, and multiple allocators under the same logical segment name are selected without NUMA-aware ordering.

The proposal makes each Master-allocatable DRAM range belong to at most one NUMA locality domain, propagates optional locality metadata during mount/remount and snapshot recovery, and adds a composable locality preference before the existing placement policy. The behavior remains backward compatible and best effort: segments or requests without NUMA metadata continue to use the current placement behavior.

This implements the Store V3 roadmap item "DRAM Adaptation: support DRAM, including NUMA affinity" in #1035.

## Motivation

On multi-socket hosts, remote-NUMA memory access can consume inter-socket bandwidth and reduce effective NIC/GPU transfer throughput. The current implementation has useful building blocks but loses locality at the Store control-plane boundary:

1. TENT discovers CPU, NIC, and GPU NUMA relationships.
2. `RealClient` can allocate one contiguous VMA divided into NUMA-bound regions and register it using the `segments::` location format.
3. Transfer Engine can resolve the concrete `cpu:N` location for an offset.
4. `Client::MountSegment()` sends only `{id, name, base, size, te_endpoint, protocol, host_id}` to the Master.
5. Master placement can prefer a host through `host_id`, but cannot select a concrete NUMA-local allocation range within that host.

As a result, Store allocation can choose an arbitrary allocator within a same-name group, and an allocation may be placed in or span a region whose locality is not the preferred one for the workload.

## Goals

- Represent DRAM locality as an explicit, immutable property of a Master allocation target.
- Ensure each NUMA-aware allocation target covers one homogeneous NUMA range.
- Allow callers to provide an optional best-effort NUMA locality hint.
- Compose locality ordering with existing placement policies rather than adding another mutually exclusive strategy.
- Preserve explicit preferred/excluded segment behavior and multi-replica failure-domain guarantees.
- Preserve compatibility with old clients, old snapshots, and segments whose locality is unknown.
- Expose metrics needed to evaluate local placement and cross-NUMA fallback.

## Non-goals

- Redesign TENT topology discovery or RDMA rail selection.
- Make NUMA locality a hard requirement by default.
- Change read-side replica selection; topology-aware reads are tracked separately by #2516.
- Introduce cross-host topology distance or fabric-cost routing.
- Require one RDMA memory registration per NUMA range.
- Rework allocator fragmentation behavior.

## Current architecture

```text
TENT topology discovery
-> NIC/GPU/CPU NUMA relationships

RealClient
-> discovers NIC NUMA nodes
-> mmap contiguous Store memory
-> mbind each region to a NUMA node
-> registerLocalMemory(..., "segments::")

Client::MountSegment
-> sends base/size/protocol/host_id
-> NUMA range information is not sent to Master

Master placement
-> selects by logical segment name / host
-> allocator inside the group is not NUMA-aware
```

## Proposed model

### 1. Introduce a stable Store locality type

The Store protocol should expose semantic locality rather than forwarding TENT's internal location grammar.

```cpp
struct MemoryLocality {
int32_t numa_node{-1}; // -1 means unknown / not applicable
};
```

A mounted allocation target has exactly one locality domain. The locality is immutable for the lifetime of that mount and must match during remount.

The initial version intentionally carries only `numa_node`. NIC/GPU identifiers and distance matrices remain owned by TENT and can be added later through compatible fields if needed.

### 2. Make NUMA regions independently allocatable

A compound `segments::n0,n1,...` Transfer Engine registration may remain one underlying memory registration, but Store should expose each NUMA-bound subrange as an independent allocation target:

```text
one TE registration
[ NUMA 0 range ][ NUMA 1 range ]
| |
+-- target A +-- target B
numa=0 numa=1
```

Each range has its own stable Segment UUID, base, size, allocator, and locality. The ranges may retain a shared user-visible placement group/host identity.

This avoids teaching `OffsetAllocator` about NUMA subranges and prevents an individual allocation from crossing a NUMA boundary.

The exact internal representation should align with the `SegmentPool` / `PlacementIndex` refactor in #3360 and #3529. Placement must rank concrete allocation targets, while the existing logical segment name remains available for compatibility with `preferred_segments` and administrative APIs.

### 3. Add an optional request locality hint

Extend the write configuration with a backward-compatible optional hint:

```cpp
struct MemoryLocalityHint {
int32_t preferred_numa_node{-1};
bool strict{false};
};
```

Phase 1 exposes this as an explicit hint. Automatic inference from source slices may be considered later because one object can contain slices from different memory or device domains.

`strict=false` is the default and preserves best-effort behavior. `strict=true` is reserved for callers that would rather fail than cross the NUMA boundary; whether strict mode belongs in the first implementation is an open question.

### 4. Compose locality with existing placement policies

NUMA locality should be a candidate-ordering layer, not a new exclusive `AllocationStrategyType`.

After explicit preferred/excluded constraints, candidates are partitioned into tiers:

```text
Tier 0: requested host + requested NUMA node
Tier 1: requested host + unknown NUMA locality
Tier 2: requested host + another NUMA node
Tier 3: another host
```

Within each tier, the configured placement policy continues to rank candidates:

```text
random
free_ratio_first
size_class_aware
fragmentation-aware, if adopted
future policies
```

This keeps locality orthogonal to utilization and fragmentation policy.

Explicit `preferred_segments` remain first. Multi-replica requests must continue to avoid placing two replicas in the same required failure domain; NUMA affinity must not weaken host-level replica separation.

### 5. Fallback semantics

The default behavior is best effort:

```text
preferred NUMA target allocation fails
-> try another target in the same NUMA domain
-> try unknown-locality targets on the same host
-> try another NUMA node on the same host
-> use the existing remote-host fallback
```

Missing, stale, or unsupported locality metadata must never make an otherwise valid allocation fail unless an explicit strict mode is selected.

### 6. Compatibility and recovery

- Old clients mount segments with `numa_node = -1` and retain current behavior.
- New clients talking to a compatible Master send locality through an optional/versioned field.
- Snapshot encoding appends optional locality fields; readers continue accepting the existing 8/9-element mounted-segment forms.
- Snapshot restore reconstructs the same allocation-domain boundaries and locality metadata.
- Remount validates immutable `{base, size, protocol, host_id, locality}` identity when locality is present.
- No object metadata or replica descriptor format change is required.

The wire-level representation must use the repository's compatible-field mechanism rather than relying on unconditional aggregate-field extension.

## Placement state and concurrency

Locality metadata is immutable after mount, so placement reads require no topology update synchronization. Allocation usage remains dynamic and is supplied by the existing allocator/placement-policy path.

Topology changes such as NIC replacement do not rewrite Segment NUMA ownership. TENT may change rail selection independently; changing the physical memory binding requires draining and remounting the affected allocation target.

## Observability

Add metrics for:

- mounted DRAM capacity by host and NUMA node
- allocated bytes by host and NUMA node
- exact-NUMA placement count and bytes
- same-host cross-NUMA fallback count and bytes
- remote-host fallback count and bytes
- unknown-locality segment/request count
- strict-locality allocation failures, if strict mode is included

The admin segment view should expose the optional NUMA node.

## Rollout plan

### Phase 1: Metadata and compatibility

- Add optional Segment locality metadata.
- Preserve old RPC and snapshot decoding.
- Expose locality through admin/debug output.
- Add mount/remount/snapshot compatibility tests.

### Phase 2: NUMA allocation domains

- Split a compound NUMA-bound Store mapping into independently allocatable ranges.
- Keep the underlying TE registration strategy independent.
- Ensure no allocation crosses a NUMA-domain boundary.
- Add allocator ownership and unmount tests.

### Phase 3: Placement

- Add optional request locality hints.
- Introduce locality-tier candidate ordering composed with current policies.
- Add fallback, preferred/excluded, and multi-replica tests.

### Phase 4: Validation and tuning

- Add synthetic topology tests that do not require NUMA hardware.
- Benchmark on a dual-socket host with NICs attached to different NUMA nodes.
- Measure local/remote NUMA bandwidth, transfer throughput, P50/P99 latency, CPU utilization, and fallback rate.
- Keep the feature opt-in until hardware results show a stable benefit.

## Alternatives considered

### Keep the current compound Segment and make `OffsetAllocator` range-aware

This would require one allocator to maintain multiple NUMA-local free-space pools and choose a pool per request. It couples physical topology to allocator internals and makes fragmentation policy more complex. Independent allocation targets keep the allocator unchanged.

### Encode NUMA in the segment name

Names such as `host#numa0` require no protocol field, but overload a user-facing identifier, complicate preferred-segment compatibility, and make topology parsing convention-dependent.

### Add a standalone `numa_first` allocation strategy

This would make locality mutually exclusive with free-ratio, size-class, or fragmentation-aware ranking. A composable locality tier is more useful and avoids strategy proliferation.

### Let TENT handle everything

TENT can choose rails and resolve the locality of an already selected address, but it cannot prevent the Store Master from allocating that address in an undesirable NUMA range. Placement requires Store-visible allocation domains.

## Risks

- Increasing the number of Master allocation targets may increase placement-index and snapshot size.
- SegmentPool refactoring may change the best internal seam; implementation should target the post-#3529 placement model when possible.
- A request-level NUMA hint may be misleading for objects assembled from heterogeneous slices.
- NUMA-local placement can reduce utilization if used as a hard constraint; best-effort fallback and metrics are required.
- Hardware gains depend on CPU, NIC, PCIe, and workload topology and must not be claimed from synthetic tests alone.

## Open questions

1. Should the first version support only best-effort hints, leaving strict locality for later?
2. Should the API expose only `numa_node`, or a future-proof locality-domain string?
3. Should compound TE registrations be represented as multiple Store Segments or as multiple internal placement targets under one external Segment?
4. Should callers provide the hint explicitly, or should the client infer it when all source slices resolve to one locality?
5. Should NUMA placement wait for #3529 to merge, or land metadata support first against the current SegmentManager?

## Related work

- #1035: Store V3 roadmap; DRAM adaptation with NUMA affinity.
- #1883: overall Mooncake roadmap.
- #2516: topology- and load-aware read replica selection; this RFC focuses on write allocation.
- #2633: host-aware local-first allocation; this RFC adds intra-host NUMA locality.
- #3360 and #3529: SegmentPool and placement-index refactoring.
- #2797 and #3651: fragmentation and size-class placement policies; NUMA locality composes with them instead of replacing them.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the documentation

Start by reading Client::MountSegment and the SegmentPool/PlacementIndex refactors (#3360 and #3529), then trace snapshot restore, remount validation, and current placement policies. The RFC defines done across compatibility metadata, independently allocatable NUMA ranges, locality-tier placement, fallback behavior, metrics, and synthetic tests, but its open questions require design decisions first.

Written by the indexing model from the issue text.