[RFC]: Consolidate Master Metadata Routing

### Changes proposed

# RFC: Consolidate Master Metadata Routing

## Summary

`MasterService` should keep the current tenant-isolated metadata layout:

```text
metadata_shards_[shard].tenants[tenant_id].metadata[user_key]
```

This preserves the strong tenant boundary: each tenant owns an independent `TenantState`, and duplicate `user_key` values across tenants are naturally isolated.

The proposed change is not to flatten tenant storage. Instead, it is to consolidate routing and per-object access behind a canonical object identity:

```text
ObjectIdentity{tenant_id, user_key}
-> MetadataRoute{shard, tenant_state, object_state}
-> ObjectState{metadata, processing state, tasks, group state}
```

The goal is to get most of the simplicity of a flat metadata model without weakening tenant isolation. Object operations should use one routing/accessor path and then read or mutate object-scoped state from one colocated object entry.

## Motivation

The current lookup path is `tenant_id + user_key -> shard -> TenantState -> metadata[user_key]`. This tenant layer is useful and should remain, but object-scoped state is spread across several tenant-local maps such as `metadata`, `processing_keys`, `replication_tasks`, `offloading_tasks`, `promotion_tasks`, and `group_members`.

That means a single object operation can repeat the same `user_key` lookup across multiple maps after it has already found the shard and tenant. It also makes routing logic easy to duplicate across call sites.

The master already has `ObjectIdentity` and `MetadataAccessorRO/RW`. This RFC proposes making that route/accessor layer the single way to reach object metadata and related state, instead of open-coding the route at individual call sites.

## Goals

- Preserve `MetadataShard::tenants[tenant_id]` as the isolation boundary.
- Use `ObjectIdentity{tenant_id, user_key}` as the canonical input to routing and accessors.
- Consolidate shard selection, tenant lookup, tenant creation, and per-object lookup in one route/accessor path.
- Colocate object-scoped state inside a per-object `ObjectState` so common operations avoid several tenant-local map lookups.
- Keep public APIs compatible: callers may continue passing `tenant_id` and `user_key` separately.
- Preserve single-tenant mode by normalizing requests to the `default` tenant when multi-tenant mode is disabled.
- Keep snapshot compatibility with the existing tenant-nested shape.

## Non-Goals

- Do not flatten `MetadataShard::tenants`.
- Do not remove tenant semantics from Mooncake Store.
- Do not require globally unique user keys.
- Do not redesign tenant quota policy storage.
- Do not introduce auth or access control.

## Proposed Model

Keep the shard and tenant structure:

```cpp
struct MetadataShard {
mutable SharedMutex mutex;
std::unordered_map tenants GUARDED_BY(mutex);
long disk_object_count GUARDED_BY(mutex) = 0;
};
```

Use `ObjectIdentity` as the canonical routing key:

```cpp
struct ObjectIdentity {
std::string tenant_id;
std::string user_key;
};
```

Normalize tenant IDs at the API boundary:

```cpp
ObjectIdentity MasterService::MakeObjectIdentityForRequest(
const std::string& user_key,
const std::string& tenant_id) const {
return {NormalizeRequestTenantId(tenant_id), user_key};
}
```

Inside each `TenantState`, aggregate per-object state by `user_key`:

```cpp
struct ObjectState {
std::optional metadata;
bool processing = false;
std::optional replication_task;
std::optional offloading_task;
std::optional promotion_task;
std::optional group_id;
};

struct TenantState {
std::unordered_map objects;
std::unordered_map>
group_members; // group_id -> set of user keys in this tenant
};
```

This keeps tenant isolation explicit while avoiding parallel lookups in `metadata`, `processing_keys`, `replication_tasks`, `offloading_tasks`, and `promotion_tasks`.

## Routing and Access

Object operations should use one route:

```text
request tenant_id + user_key
-> NormalizeRequestTenantId()
-> ObjectIdentity{tenant_id, user_key}
-> getMetadataShardIndex(object_id)
-> shard.tenants[object_id.tenant_id]
-> tenant_state.objects[object_id.user_key]
-> ObjectState
```

`MetadataAccessorRO/RW` should own this route. Call sites should not manually repeat shard lookup, tenant lookup, and object-state lookup unless they are doing bulk iteration.

For compatibility, the `default` tenant may keep the current `hash(user_key)` routing. Other tenants should combine tenant and key when choosing the shard.

## Tenant-Aware Behavior

Tenant-aware subsystems keep using tenant ID explicitly:

- Quota remains keyed by tenant ID. Create/delete paths take the tenant from `object_id.tenant_id`.
- Metrics continue aggregating by tenant.
- Tenant admin operations can still find and remove an entire `TenantState` per shard.
- Snapshot/restore can keep the existing tenant-nested shape.

## Migration Plan

1. Keep `MetadataShard::tenants` and `TenantState` as the storage boundary.
2. Introduce `ObjectState` inside `TenantState`.
3. Update `MetadataAccessorRO/RW` to route from `ObjectIdentity` to one `ObjectState`.
4. Move `metadata`, `processing_keys`, `replication_tasks`, `offloading_tasks`, and `promotion_tasks` into `ObjectState` fields.
5. Keep `group_members` tenant-local, but store the per-object group link in `ObjectState::group_id`.
6. Update iteration paths: quota reconciliation, eviction, snapshot, admin remove/list, and cache accounting.
7. Remove the old parallel tenant-local maps after all call sites use `ObjectState`.

## Testing Plan

- Same `user_key` under two tenants maps to two isolated `TenantState` entries.
- Single-tenant mode still normalizes to `default`.
- Object create/delete quota accounting remains correct.
- Tenant remove/list operations still work without scanning a global object map.
- Snapshot/restore preserves tenant-nested metadata.
- Group IDs are isolated per tenant.
- Replication/offloading/promotion state is found through the same `ObjectState` as metadata.
- Eviction and disk-object accounting remain correct after object deletion.

## Risks

- A partial migration could create a hybrid model where metadata is in `ObjectState` but tasks still live in parallel maps, losing the lookup reduction.
- `ObjectState` must not blur tenant boundaries; it should always live under one `TenantState`.
- Existing snapshot code may need a compatibility layer if `TenantState` serialization changes from parallel maps to `objects[user_key] -> ObjectState`.

## Conclusion

Keep the tenant-nested metadata layout for strong isolation, but make `ObjectIdentity` and `MetadataAccessorRO/RW` the only normal route into object state. Within each `TenantState`, consolidate per-object metadata, processing flags, and tasks into `ObjectState` so common object operations get a flat-access experience without giving up tenant isolation.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Start by locating MasterService and MetadataAccessorRO/RW, then review the proposed ObjectIdentity, ObjectState, and TenantState relationships. Compare the migration and testing plans against the current routing, iteration, snapshot, quota, group, and task paths; done means tenant isolation, default-tenant normalization, compatibility, accounting, and state access all remain correct.

Written by the indexing model from the issue text.