[RFC]: Lock-Free Tenant Quota Charge/Release with Unified Charged Bytes

### Changes proposed

## Changes proposed

### Summary

This RFC proposes replacing Mooncake Store's per-tenant `used_bytes` and `reserved_bytes` quota-accounting state with one atomic `charged_bytes` counter.

Memory is charged when Put, Copy, Move, or promotion admission starts. A successful operation keeps the charge, while a failed, revoked, partially completed, or expired operation refunds the unused portion. Completion no longer moves bytes from a global reserved counter to a global used counter.

The tenant quota control plane remains lock-based. Policy persistence, effective-quota recomputation, tenant registration, snapshots, and administrative mutation continue to use the existing control-plane locks. Only the high-frequency charge and release operations become independent of the quota-table shard mutex.

In this RFC, "lock-free" describes only the tenant quota charge/release primitive. A complete Put, Copy, Move, or promotion still uses its existing metadata, allocator, and task locks. Move is included because it shares the same `ReplicationTask` quota-settlement path as Copy; this RFC does not otherwise broaden the lock-free scope.

The proposed design:

- uses one compare-and-swap loop for quota admission;
- keeps operation-local `pending_charge_bytes` for exact rollback;
- stores a stable quota-account handle in each metadata `TenantState`;
- blocks new admissions through a bit in the same atomic charge state;
- preserves strict policy-deletion semantics, including zero-memory objects;
- rebuilds charged usage from authoritative metadata after restore;
- replaces the old quota API and metrics with a single charged-byte model; and
- does not provide a dual implementation, compatibility aliases, or downgrade guarantees.

### Motivation

The current quota state distinguishes:

```text
used_bytes completed memory-replica charge
reserved_bytes admitted but not yet completed memory-replica charge
```

Admission checks:

```text
used_bytes + reserved_bytes + incoming_bytes <= effective_quota_bytes
```

`Reserve`, `Commit`, `CommitAdditional`, `Abort`, `Release`, and `ReleasePartial` all acquire the tenant quota shard mutex. Although the quota table is sharded, every operation for the same tenant maps to the same shard and therefore serializes on one mutex. A large tenant writing keys distributed across many metadata shards can still be bottlenecked by its single quota mutex.

The global reserved/used distinction is not required for quota admission. Admission only needs the total number of bytes already charged to the tenant. The distinction is useful when settling one operation, but that information is already present in `ObjectMetadata`, `ReplicationTask`, and `PromotionTask`.

Combining both global counters allows admission to become:

```text
load charged bytes
check charged + incoming against effective quota
CAS charged to charged + incoming
```

Completion normally performs no aggregate byte update. Failure and partial completion only subtract the corresponding operation-local pending charge.

### Goals

1. Remove the quota-table shard mutex from charge and release hot paths.
2. Preserve strict per-tenant admission: successful concurrent admissions must not exceed the effective quota at their linearization points.
3. Preserve policy deletion, dynamic policy updates, orphan-account cleanup, zero-memory metadata, timeout cleanup, and HA restore behavior.
4. Preserve the existing peak-charge behavior of size-changing Upsert, Copy, Move, and promotion.
5. Detect accounting underflow instead of allowing unsigned wraparound.
6. Avoid introducing a lock-free map or general-purpose memory-reclamation scheme in the first implementation.
7. Remove accounting fields, APIs, and metrics that are unnecessary once backward compatibility is not required.

### Non-goals

- Making the entire Master request path lock-free.
- Removing metadata-shard, allocator, snapshot, or policy-store locks.
- Changing tenant effective-quota allocation policy.
- Changing eviction selection or retry policy.
- Persisting in-flight quota charge across Master restart or failover.
- Introducing per-metadata-shard quota credits in the first implementation.
- Providing a transactionally consistent multi-tenant metrics snapshot.
- Supporting mixed-version Masters, rolling downgrade, or legacy quota API and metric consumers.

## Terminology

| Term | Meaning |
|------|---------|
| Requested quota | The configured quota requested by the tenant policy. |
| Effective quota | The tenant's current quota after cluster-capacity scaling. |
| Charged bytes | All bytes currently counted against admission, including committed and in-flight memory replicas. |
| Pending charge | The portion owned by one in-flight operation and refundable on failure. |
| Committed charge | The portion owned by completed memory replicas in object metadata. |
| Admission closed | The quota account rejects new positive and zero-byte admissions. Releases remain allowed. |
| Orphan account | An account with metadata or charge but no explicit tenant policy. |

`charged_bytes` is an accounting term. A charge can be installed immediately before physical allocation and refunded if allocation fails. It does not mean that every charged byte is already readable or physically allocated at the instant a metrics request observes it.

## Current behavior

The current quota state machine performs the following transitions:

```text
Reserve(N): reserved += N
Commit(N): reserved -= N; used += N
CommitAdditional(N): reserved -= N; used += N
Abort(N): reserved -= N
Release(N): used -= N
ReleasePartial(N): used -= N
```

The quota table lock provides three different kinds of protection:

1. atomic check-and-increment during admission;
2. consistent movement between multiple accounting fields; and
3. tenant entry and policy lifetime protection.

Removing `reserved_bytes` eliminates the second requirement, but it does not by itself solve policy deletion or tenant-entry lifetime. The proposed design addresses those requirements explicitly.

## Proposed semantics

### Unified charge

The tenant aggregate becomes:

```text
charged_bytes = committed memory charge + in-flight pending memory charge
```

The state transitions become:

```text
TryCharge(N): CAS charged += N
SettleSuccess(pending, used): charged -= pending - used
RefundPending(N): charged -= N
ReleaseCommitted(N): charged -= N
```

PutStart therefore has the same admission behavior as the current `Reserve`. The visible difference is that the aggregate counter is immediately reported as charged rather than split between used and reserved.

### Breaking API and metric changes

This RFC is an intentional breaking refactor. The implementation removes the old tenant-ID-based `Reserve`, `Commit`, `CommitAdditional`, `Abort`, `Release`, and `ReleasePartial` accounting API and replaces it with handle-based `TryCharge` and `Release` plus operation-settlement helpers.

The admin response and Prometheus surface remove `used_bytes`, `reserved_bytes`, `committed_count`, and `metadata_object_count`. They expose `charged_bytes`, effective and requested quota, and admission state directly. No field aliases, zero-valued placeholders, dual accounting mode, or compatibility feature flag are provided.

The change must therefore be deployed as one control-plane cutover rather than as a mixed-version rolling upgrade.

### Operation-local pending charge

This RFC removes `reserved_bytes` from the tenant quota account. It does not remove knowledge of the charge owned by an in-flight operation.

The existing fields are renamed:

```text
reserved_quota_charge_bytes -> pending_quota_charge_bytes
```

They remain in:

- `ObjectMetadata`;
- `ReplicationTask`; and
- `PromotionTask`.

These fields are protected by the existing metadata-shard lock. End, Revoke, cleanup, and durable-finalization paths consume them exactly once, preferably through a centralized helper using `std::exchange(field, 0)`.

Without operation-local pending charge, an aborting path cannot determine the correct refund after partial completion, replica loss, Upsert replacement, or promotion timeout.

## Quota account representation

### Atomic charge state

The preferred representation is:

```cpp
class TenantQuotaAccount {
public:
static constexpr uint64_t kAdmissionClosed = 1ULL << 63;
static constexpr uint64_t kChargedBytesMask = kAdmissionClosed - 1;
static constexpr uint64_t kMaxChargedBytes = kChargedBytesMask;

private:
// Bit 63: admission is closed.
// Bits 0..62: charged bytes.
alignas(64) std::atomic charged_state_{
kAdmissionClosed};

std::atomic effective_quota_bytes_{0};
std::atomic policy_sequence_{0};

// Accessed only under control-plane locks.
uint64_t requested_quota_bytes_{0};
bool has_explicit_policy_{false};
};
```

Encoding the admission gate and byte count in one atomic word gives policy deletion and admission a single, exact ordering point.

This reserves the high bit and limits one tenant to `2^63 - 1` charged bytes, approximately 8 EiB. The limit is far above practical Mooncake deployments, but it is an observable validation change and must be documented. Policy input, capacity recomputation, requested replica charge, and rebuild code must reject values above this limit rather than truncate them.

### Policy sequence

`policy_sequence_` is even while the policy is stable and odd while an effective quota or registration state is being changed.

A charge operation records the sequence before its CAS and checks it again afterward. If the sequence changed, the operation refunds its charge and retries against the new policy.

This prevents an old request from succeeding across:

- effective-quota shrink;
- connector policy removal;
- policy delete followed by re-registration; or
- disabling and later re-enabling an orphan account.

Only control-plane updates write the sequence, so it is not a contended data-plane CAS location.

### Which fields require atomic operations

Only `charged_state_` needs a compare-and-swap loop for quota correctness. The other fields do not form one atomic quota equation:

| Field | Data-plane access | Reason |
|-------|-------------------|--------|
| `charged_state_` | CAS for charge, release, and admission close | The limit check and byte update must linearize together; the gate must not race policy deletion. |
| `effective_quota_bytes_` | Atomic load | The control plane stores it under existing locks; charge reads it without taking those locks. |
| `policy_sequence_` | Two atomic loads per charge; control-plane increment on update | Detects policy-generation changes and ABA without joining the hot CAS cache line. |
| Requested quota, registration flags, account registry | Existing control-plane locks | These paths are administrative and do not justify a lock-free registry. |

`committed_count` and `metadata_object_count` are removed rather than made atomic. They are not required for admission, release, or policy deletion when the control plane can perform an authoritative metadata scan.

The hot-path quota interface is intentionally small:

```cpp
TenantQuotaChargeResult TryCharge(TenantQuotaHandle account,
uint64_t bytes);

TenantQuotaResult Release(TenantQuotaHandle account,
uint64_t bytes);
```

Neither method accepts a tenant ID or accesses the account registry. Tenant-ID-based lookup, policy upsert/delete, effective-quota recomputation, listing, and admin snapshots remain control-plane interfaces protected by the existing locks.

### Stable account lifetime

Quota accounts are stored behind stable pointers:

```cpp
using TenantQuotaHandle = TenantQuotaAccount*;

std::map> accounts_;
```

An account is not freed or erased while the Master is running. Removing a policy leaves a closed tombstone. This avoids RCU, hazard pointers, and shared-pointer reference-count traffic in charge/release paths.

Each metadata-shard `TenantState` stores the handle:

```cpp
struct TenantState {
TenantQuotaHandle quota_account{nullptr};
// Existing metadata and task maps.
};
```

All `TenantState` instances for the same tenant point to the same quota account. A normal write obtains the handle while validating the tenant policy and binds it when creating the per-shard `TenantState`. Existing-object paths reuse the bound handle and do not perform a quota-map lookup.

Creating the first `TenantState` for a tenant in one metadata shard may take the existing quota-registry shard lock to resolve or create the stable handle. This is a cold lifetime event, not part of `TryCharge` or `Release`. If the same workload repeatedly removes and recreates otherwise-empty `TenantState`s, that lookup may still appear in profiles; avoiding it would require retaining per-shard tombstones or a lock-free registry and is outside this RFC.

After a charge succeeds, every owner that may refund it must retain access to the same handle. Normally the owning `TenantState` provides it. Any deferred callback or cleanup record that can outlive removal of that `TenantState` must capture `TenantQuotaHandle` directly. Hot settlement code must not fall back to a tenant-ID registry lookup.

Tombstone memory grows with the number of distinct tenant IDs seen during the Master process lifetime. The first implementation accepts this tradeoff. Bounded reclamation can be added later if high-rate tenant churn becomes a real deployment requirement.

## Admission algorithm

The charge result includes the deficit observed by the failing attempt:

```cpp
struct TenantQuotaChargeFailure {
TenantQuotaError error;
uint64_t deficit_bytes{0};
};

using TenantQuotaChargeResult =
tl::expected;
```

At a high level, `TryCharge(bytes)` performs:

```text
loop:
read stable policy sequence
read charged state
reject if admission is closed
read effective quota
reject if charged + incoming exceeds quota
CAS charged state
verify policy sequence and admission gate
if policy changed:
refund and retry
return success
```

The overflow-safe admission check is:

```cpp
if (current > limit || incoming > limit - current) {
return quota_exceeded;
}
```

Successful CAS is the admission linearization point.

Illustrative pseudocode, with error plumbing omitted, is:

```cpp
for (;;) {
const uint64_t sequence_before =
account->policy_sequence_.load(std::memory_order_acquire);
if (sequence_before & 1) {
continue;
}

uint64_t expected =
account->charged_state_.load(std::memory_order_acquire);
if (expected & kAdmissionClosed) {
return TENANT_NOT_REGISTERED;
}

const uint64_t charged = expected & kChargedBytesMask;
const uint64_t limit =
account->effective_quota_bytes_.load(std::memory_order_acquire);

if (bytes != 0 &&
(charged > limit || bytes > limit - charged)) {
return QuotaExceeded(Deficit(charged, bytes, limit));
}

if (bytes != 0) {
const uint64_t desired = charged + bytes;
if (!account->charged_state_.compare_exchange_weak(
expected, desired, std::memory_order_acq_rel,
std::memory_order_acquire)) {
continue;
}
}

const uint64_t state_after =
account->charged_state_.load(std::memory_order_acquire);
const uint64_t sequence_after =
account->policy_sequence_.load(std::memory_order_acquire);

if (sequence_before == sequence_after &&
!(sequence_after & 1) &&
!(state_after & kAdmissionClosed)) {
return Success();
}

if (bytes != 0) {
Release(account, bytes);
}
// Retry to return the result of the new policy generation.
}
```

Every control-plane change that can alter registration or admission semantics must bracket its atomic stores and admission-gate mutation by changing `policy_sequence_` from even to odd and back to even. Control-plane writers are already serialized by the existing locks.

For a registered tenant, a zero-byte admission continues to succeed even when the account is over quota. It still verifies that admission is open and that the policy sequence is stable.

Returning the deficit from the failed charge attempt replaces the current separate `ComputeDeficit` lookup in the Put eviction/retry path. The deficit is a point-in-time eviction target; concurrent release can make it conservative, which is already acceptable to the retry loop.

### Release algorithm

Release must work even when admission is closed, because orphaned or administratively disabled accounts still need to drain.

```text
loop:
load charged state
extract charged byte count
reject if refund exceeds charged bytes
preserve admission bit
CAS charged state to charged - refund
```

A CAS loop is preferred over unchecked `fetch_sub` so an accounting bug cannot underflow the byte field and corrupt the admission bit.

Illustrative release pseudocode is:

```cpp
uint64_t expected =
account->charged_state_.load(std::memory_order_acquire);
for (;;) {
const uint64_t charged = expected & kChargedBytesMask;
if (bytes > charged) {
return ACCOUNTING_UNDERFLOW;
}

const uint64_t desired =
(expected & kAdmissionClosed) | (charged - bytes);
if (account->charged_state_.compare_exchange_weak(
expected, desired, std::memory_order_acq_rel,
std::memory_order_acquire)) {
return Success();
}
}
```

No registration or effective-quota check is performed during release.

## Control-plane behavior

The existing control-plane locking remains:

- `tenant_quota_policy_mutex_` serializes policy persistence and tenant registration changes;
- `tenant_quota_recompute_mutex_` serializes the capacity snapshot with effective-quota recomputation;
- quota-table shard locks protect account registry and control-only fields; and
- the quota-table recompute mutex protects cross-shard policy assignment.

Charge and release methods accept a `TenantQuotaHandle` and do not acquire any of these quota-table locks.

### Effective-quota update

An effective quota update uses a sequence write:

```text
increment sequence to odd
store effective quota
increment sequence to even
```

An admission completed before the odd sequence begins can be linearized before the update. An admission overlapping the update detects the sequence change, refunds, and retries.

### Delete policy if empty

Policy deletion is a control-plane operation and may take the existing policy and metadata locks. It does not need data-plane counters.

Under `tenant_quota_policy_mutex_`, deletion:

1. opens a short odd-sequence update window;
2. closes admission only if the account has zero charged bytes;
3. completes the sequence update;
4. calls the authoritative `TenantHasObjects(tenant_id)` metadata scan while admission remains closed;
5. persists policy removal; and
6. updates the control-only policy fields while leaving a closed tombstone.

Admission close uses:

```cpp
uint64_t expected = 0;
charged_state_.compare_exchange_strong(
expected, kAdmissionClosed);
```

There are only two outcomes:

1. charge wins first, so delete observes nonzero charge and returns `TENANT_NOT_EMPTY`; or
2. close wins first, so the charge observes the closed bit and returns `TENANT_NOT_REGISTERED`.

If `TenantHasObjects` finds any metadata, or if policy persistence fails, deletion opens another short sequence window, restores the policy, reopens admission, completes the sequence update, and returns an error. The scan holds `tenant_quota_policy_mutex_` and visits metadata shards one at a time, consistent with the existing lock order.

Odd sequence windows contain only in-memory atomic and policy-field updates. Connector I/O and metadata scans run with admission closed but the sequence even, so `TryCharge` returns `TENANT_NOT_REGISTERED` instead of spinning for the duration of a slow control-plane operation.

### Connector policy removal with existing objects

Applying a connector snapshot can remove an explicit policy while metadata still exists. This creates an orphan account rather than deleting its state.

The control plane atomically sets the admission-closed bit while preserving the low charged-byte bits. Existing charges remain visible and releasable, but new admissions fail. Re-registering the tenant updates its effective quota, advances the policy sequence, and reopens admission.

## Zero-charge metadata

NOF-only and local-disk-only metadata may have zero memory quota charge. `charged_bytes` therefore cannot by itself prevent policy deletion from racing with creation of such metadata.

The first implementation retains the current slow-path rule:

- creation of new zero-charge metadata holds `tenant_quota_policy_mutex_` until metadata insertion completes; and
- policy deletion holds the same mutex.

Policy deletion closes admission and scans authoritative metadata while holding the same outer lock. It therefore cannot miss a concurrent zero-charge creation. This RFC does not add an object counter or a data-plane reader counter solely to accelerate this cold administrative path.

Every metadata-creation path must satisfy one of two publication rules:

1. a positive charge remains installed from successful admission until the new metadata is visible; or
2. `tenant_quota_policy_mutex_` remains held from policy validation until the new metadata is visible.

A path that initially charges positive bytes but refunds the charge to zero before publishing metadata must use the second rule. It must acquire the policy mutex before the metadata-shard lock, or drop the inner lock and retry through the protected creation path; it must not invert the existing lock order. This prevents policy deletion from closing admission, observing an empty tenant, and then being followed by publication of uncharged metadata.

## Operation settlement

### Put and new-object Upsert

PutStart:

1. Compute the maximum requested memory-replica charge.
2. Call `TryCharge(requested)`.
3. Allocate replicas.
4. On allocation or metadata-insertion failure, refund the full charge.
5. Store the successful amount in `ObjectMetadata::pending_quota_charge_bytes`.

PutEnd:

1. Compute charge for completed memory replicas.
2. Compute the newly committed portion relative to `committed_quota_charge_bytes`.
3. Refund `pending - newly_committed`.
4. Clear pending charge.
5. Update committed charge.

Completion does not add newly committed bytes to the tenant aggregate; they were already charged at Start.

PutRevoke, stale-processing cleanup, and object erase refund the remaining pending amount exactly once.

### Same-size Upsert

Same-size Upsert reuses existing memory buffers and does not add quota charge. The existing committed charge remains installed while replicas temporarily move from COMPLETE to PROCESSING.

### Size-changing Upsert

Size-changing Upsert preserves the current peak-accounting behavior:

```text
charged during transfer = old committed charge + new pending charge
```

If new allocation fails after old metadata has been removed, both the failed new charge and the no-longer-owned old charge are released.

If the replacement succeeds, PutEnd:

- settles the new pending charge;
- releases `pending_replaced_quota_charge_bytes`; and
- clears replacement state.

### Copy

CopyStart charges:

```text
object_size * number_of_new_memory_targets
```

CopyEnd keeps the charge for successfully completed targets and refunds failed or missing targets. It adds the successful portion to the object's committed charge but does not modify aggregate charged bytes again.

CopyRevoke and replication-task expiry refund the full remaining pending charge.

### Move

MoveStart temporarily charges the target allocation while the source remains charged. On successful completion, the source's existing committed charge becomes ownership of the target and the entire temporary pending charge is refunded.

The object's total committed charge therefore remains unchanged after a same-sized Move.

Move failure, revoke, and expiry also refund the temporary pending charge.

### Promotion

Promotion allocation charges one object size before allocating a MEMORY replica.

On success, the pending charge becomes committed without another aggregate increment. On failure, notification loss, replica loss, or reaper expiry, the pending charge is refunded.

### Replica removal and eviction

Partial replica removal computes completed memory charge before and after the metadata mutation. The difference is released from both the object's committed charge and the tenant aggregate.

No tenant-level committed-object counter is updated. The object's `committed_quota_charge_bytes` remains the authoritative ownership record used for subsequent release and reconciliation.

### Centralized settlement helpers

The implementation should centralize settlement rather than reproduce charge arithmetic across every error path:

```cpp
void RefundPendingCharge(TenantState&, uint64_t& pending);

void SettleInitialCharge(TenantState&, ObjectMetadata&,
uint64_t actual_charge);

void SettleAdditionalCharge(TenantState&, ObjectMetadata&,
uint64_t& task_pending,
uint64_t actual_charge);
```

All Put, Copy, Move, promotion, erase, durable-finalize, and reaper paths must use these helpers.

## Snapshot and metrics semantics

### Snapshot consistency

The replacement admin snapshot is:

```cpp
struct TenantQuotaSnapshot {
TenantId tenant_id;
uint64_t requested_quota_bytes;
uint64_t effective_quota_bytes;
uint64_t charged_bytes;
bool admission_closed;
bool has_explicit_policy;
bool over_quota;
};
```

Quota snapshots atomically load:

- charged bytes;
- effective quota; and
- admission-closed state.

Requested quota and explicit-policy state are read under the existing control-plane lock. The resulting snapshot is point-in-time approximate across fields. This is sufficient for admin and Prometheus output. Policy deletion does not rely on this snapshot; it uses the admission gate and authoritative metadata scan.

`over_quota` becomes a derived value:

```text
charged_bytes > effective_quota_bytes
```

An orphan account is represented explicitly by `has_explicit_policy == false` and `admission_closed == true`, including when its objects have zero memory charge.

### Breaking metrics replacement

Add:

```text
mooncake_tenant_quota_charged_bytes
mooncake_tenant_quota_admission_closed
```

Remove the old used, reserved, committed-count, and metadata-object-count quota metrics and response fields in the same change. Dashboards and clients must move directly to `charged_bytes` and `admission_closed`.

The old used/reserved split is intentionally not reconstructed. If operators later need in-flight visibility, it should be added as telemetry derived from operation state and must not become a second quota source of truth.

## HA, snapshot restore, and rebuild

`charged_state_`, quota handles, and operation-local pending charge are derived runtime state and are not added to the snapshot or oplog wire format.

Startup remains:

```text
restore metadata
load tenant policies
rebuild quota usage from metadata
start serving requests
```

Rebuild:

1. creates or finds the stable account for every tenant;
2. binds each restored `TenantState` to the account;
3. clears pending and replaced charge;
4. recomputes committed charge from completed MEMORY replicas;
5. rebuilds charged bytes; and
6. marks tenants without a loaded policy as closed orphan accounts.

Rebuild must run while request processing is quiescent. Runtime code must not overwrite `charged_state_` while charge or release operations are active.

All metadata-creation paths must bind a quota handle, including:

- normal Put and Upsert;
- `MetadataAccessorRW::Create`;
- snapshot deserialization;
- standby restore;
- oplog application; and
- creation of local-disk-only metadata.

Quota accounting remains derived from metadata, so the new runtime counters are not serialized. However, this RFC does not guarantee mixed-version operation or downgrade: the quota admin API, metrics, and internal interfaces change in one cutover.

## Code structure

| File | Proposed change |
|------|-----------------|
| `mooncake-store/include/tenant_quota.h` | Replace the old accounting API and state with `TenantQuotaAccount`, handle, charge-result, and minimal snapshot types. |
| `mooncake-store/src/tenant_quota.cpp` | Implement CAS admission, guarded release, policy gate, minimal snapshots, and rebuild; remove used/reserved and count bookkeeping. |
| `mooncake-store/include/tenant_quota_sharded.h` | Retain control-plane sharding and stable account registry; expose handle-based hot APIs. |
| `mooncake-store/include/tenant_quota_sharded_impl.h` | Delete tenant-ID-based hot accounting methods; keep locks only for policy, list, handle lookup, and recompute. |
| `mooncake-store/include/master_service.h` | Bind quota handles to `TenantState`; replace reserved fields with pending charge ownership; add settlement helpers. |
| `mooncake-store/src/master_service.cpp` | Migrate Put, Upsert, Copy, Move, promotion, erase, eviction, reaper, durable-finalize, and rebuild paths; remove count updates. |
| `mooncake-store/src/master_admin_service.cpp` | Replace used/reserved and count output with charged bytes and admission state. |
| `mooncake-store/tests/tenant_quota_test.cpp` | Add atomic-accounting and policy-race tests. |
| `mooncake-store/tests/master_service_tenant_quota_test.cpp` | Add operation-settlement, zero-charge, policy, and HA tests. |

## Correctness invariants

The implementation must maintain:

1. The low bits of `charged_state_` never exceed `kMaxChargedBytes`.
2. A successful positive admission has one CAS linearization point.
3. Policy deletion and a new positive admission cannot both succeed.
4. Admission-closed accounts reject charge but accept release.
5. Every pending charge has exactly one owner in metadata or a task.
6. Every pending charge is consumed exactly once by success, failure, revoke, erase, durable finalization, or reaper expiry.
7. Object committed charge equals charge for its completed MEMORY replicas after an operation settles.
8. After the system becomes quiescent:

```text
tenant charged bytes
= sum(object committed charge)
+ sum(live operation pending charge)
```

9. After restart or failover, pending charge is zero and charged bytes equal completed MEMORY charge reconstructed from restored metadata.
10. Zero-charge metadata prevents successful policy deletion.
11. New metadata is published only while protected by a positive charge or `tenant_quota_policy_mutex_`.

## Testing plan

### Quota-account unit tests

- Concurrent charge admits exactly the effective quota.
- Concurrent charge and release never underflow or exceed the admitted total.
- Charge overflow is rejected without wrapping.
- Release greater than charged bytes reports accounting mismatch.
- Release works while admission is closed.
- Policy shrink racing charge either orders before the charge or causes retry.
- Policy delete racing charge cannot allow both operations to succeed.
- Closed orphan accounts drain to zero.
- Disable/re-enable does not allow an old admission to cross a policy generation.
- `2^63 - 1` boundary behavior is explicit.
- Snapshot reads correctly mask the admission bit.
- Minimal snapshots contain no used/reserved or object-count compatibility fields.

### MasterService tests

- PutStart immediately increases charged bytes.
- PutEnd does not double-charge and refunds unused replica charge.
- Allocation and metadata-insertion failures refund the full amount.
- PutRevoke and processing timeout refund exactly once.
- Same-size Upsert preserves charged bytes.
- Size-changing Upsert charges old and new allocations concurrently.
- Size-changing Upsert success and failure settle both ownerships correctly.
- Copy partial success retains only completed target charge.
- Move success leaves total committed charge unchanged.
- Promotion success, failure, client notification loss, and timeout settle correctly.
- Replica eviction, segment loss, stale-handle cleanup, and object removal release committed charge.
- Zero-charge Put and policy delete remain serialized without metadata counters.
- A creation that can refund its positive charge before metadata publication acquires policy-mutex protection without inverting lock order.
- Policy deletion closes admission, finds zero-charge metadata through `TenantHasObjects`, and reopens admission when deletion is rejected.
- Connector-save failure reopens admission.
- Policy removal leaves a releasable orphan.
- Restore rebuilds charge from completed replicas and drops pending charge.
- Durable callbacks and reapers cannot double-refund.
- Admin responses and Prometheus output expose only the new quota model.

### Concurrency and performance validation

- Run quota and policy race tests under ThreadSanitizer.
- Add a benchmark with one tenant writing keys across many metadata shards.
- Compare throughput and tail latency against the shard-mutex implementation.
- Record CAS retries to identify pathological contention or policy-update retry loops.
- Verify from profiling that quota shard mutex acquisition is absent from charge and release call stacks.

## Implementation and cutover plan

The work can be split into reviewable commits, but the final branch is deployed as one breaking control-plane cutover:

1. **Quota account core**
- add the stable account and atomic charged state;
- add unit and race tests.
2. **Master hot-path migration**
- bind handles;
- migrate operation settlement;
- migrate reaper and HA rebuild;
- remove hot-path quota shard locking.
3. **Breaking cleanup**
- delete used/reserved and tenant-level count state;
- delete tenant-ID-based legacy accounting methods;
- replace admin responses and metrics;
- add performance benchmarks.

A test-only reconciliation pass should compare atomic charged bytes against:

```text
sum(committed metadata charge) + sum(pending operation charge)
```

There is no dual accounting implementation, compatibility feature flag, or mixed-version mode. Operators must update quota API consumers and dashboards as part of the same deployment.

## Performance expectations

For one tenant, the existing implementation serializes quota updates through a mutex. The proposed design replaces that critical section with a CAS loop over one cache line.

This removes scheduler blocking and mutex handoff, but it does not remove cache-line contention. At very high core counts, a single tenant's exact global quota is inherently a shared synchronization point.

If the charged-state cache line remains a bottleneck after this change, a future RFC can consider distributing quota credits to metadata shards and refilling them from a global account in batches. That design has more complex fairness, reclamation, and policy-update semantics and is intentionally not part of this RFC.

## Alternatives considered

### Keep reserved bytes and add an atomic total

An atomic `used + reserved` total could drive admission while the old counters remain for observability. This reduces semantic change, but every operation must update multiple counters and exact snapshots still require locking or a sequence protocol. It also retains two sources of truth for the same quota.

### Make used and reserved independently atomic

Admission cannot atomically check and update the sum of two independent counters. Concurrent commits and reservations can observe inconsistent pairs, so another combined counter or a 128-bit atomic is still required.

### Keep tenant object and committed counters

Tenant-level counters make policy deletion and some metrics O(1), but every object creation, removal, and committed-charge transition performs another cross-shard atomic update. Because policy deletion is a cold administrative path and already performs an authoritative metadata check, this RFC removes the counters and pays the scan cost there.

### Separate atomic admission gate and reader counter

Policy deletion could close a separate gate and wait for active admission readers to drain. This preserves the full 64-bit byte range, but adds at least two additional atomic operations to every charge attempt and introduces a second contended cache line.

### 128-bit compare-and-swap

A 128-bit state can hold the full byte count, registration state, and policy generation. Lock-free 128-bit atomics are not uniformly guaranteed across Mooncake's supported platforms and toolchains; some implementations fall back to a library lock. The proposed 64-bit state has more predictable behavior.

### Keep the quota shard mutex

This remains the simplest correctness model and may be sufficient for tenants with low write concurrency. It does not address the target workload where one tenant writes across many metadata shards.

### Lock-free tenant registry with reclamation

RCU or hazard-pointer lookup would also remove the policy lookup lock. The selected scope removes locking only from byte charge/release and uses stable tombstones for lifetime. A general lock-free registry is not necessary for the expected benefit.

### Per-shard quota credits

Local credits can reduce cache-line contention further, but require reclaiming unused credits during policy shrink, segment changes, tenant deletion, and failover. CAS on the exact tenant total is a substantially smaller first step.

## Risks and mitigations

| Risk | Mitigation |
|------|------------|
| Missing refund on a rare error path | Central settlement helpers, reconciliation tests, and reaper/HA coverage. |
| Double refund | Consume operation-local pending charge exactly once under the metadata shard lock. |
| Policy ABA | Stable accounts plus `policy_sequence_` validation. |
| Use-after-free | Never erase quota accounts during process lifetime. |
| Tombstone growth | Track account count; add bounded reclamation only if tenant churn requires it. |
| New 8 EiB per-tenant limit | Validate explicitly and document; consider 128-bit state only if a real deployment requires it. |
| Breaking admin API and metrics | Announce the cutover and update in-tree dashboards and clients in the same change. |
| Slow policy deletion scan | Accept the authoritative metadata scan on this cold control-plane path; admission and release remain unaffected. |
| CAS contention remains high | Measure retries; consider per-shard credits in a later RFC. |
| Runtime rebuild races admission | Restrict rebuild to initialization/failover quiescence. |

## Feedback requested

This RFC especially requests feedback on:

- whether charging at Start and reporting the amount as `charged_bytes` is the right operator-facing semantic;
- reserving the high bit of a 64-bit charge word and introducing an explicit `2^63 - 1` per-tenant limit;
- keeping operation-local pending charge while removing global reserved state;
- stable process-lifetime tombstones versus adding account reclamation;
- removing tenant-level object and committed counts in favor of a policy-deletion metadata scan;
- preserving the existing policy mutex only for zero-charge metadata creation;
- accepting a direct breaking replacement of the quota API and metrics;
- whether strict policy-sequence retry is preferable to allowing overlapping admissions to linearize before a quota update; and
- whether the expected same-tenant write concurrency justifies replacing the current shard mutex before considering per-shard quota credits.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Start by locating TenantQuotaAccount and the ObjectMetadata, ReplicationTask, and PromotionTask definitions mentioned in the RFC, then trace the existing Reserve, Commit, Abort, Release, and policy-deletion paths. Done means the unified charged_bytes and operation-local pending-charge semantics replace the old accounting API while preserving admission, rollback, restore, and policy-lifetime behavior; the RFC does not name specific tests.

Written by the indexing model from the issue text.