kvcache-ai / kvcache-ai/Mooncake
[Bug]: Over-quota tenants silently lose writes — RFC #2153's rejection path is unreachable because eviction always succeeds
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report
With `-enable_multi_tenants=true` on `0.3.13`, a tenant that exceeds its effective quota gets
**`rc=0` on every write while the master discards the data on arrival**. No rejection, no
`over_quota` flag, nothing in the logs, and nothing in the global eviction counters. From the
caller's side it is indistinguishable from a successful write followed by a cache miss.
RFC #2153, which this feature implements, specifies the opposite:
> If this fails, Mooncake may run tenant-scoped eviction for that tenant. **If eviction cannot
> create enough headroom, admission fails with a tenant quota error.**
> On memory shrink: **New writes are rejected until usage returns below effective quota.**
We think the rejection path is not merely unused but **structurally unreachable**: tenant-scoped
eviction picks victims from the same tenant, and a tenant that is actively writing always has
recently-written evictable objects. So "eviction cannot create enough headroom" never becomes
true, and admission never fails — eviction just consumes the tenant's own new data.
### Reproduction
Pool: 600 GiB across 3 store nodes. Two tenants registered through
`PUT /api/v1/tenant_quotas`:
```
default requested = 1024 TiB effective = 599.99 GiB
probe2 requested = 10 GiB effective = 5.9 MB
```
(The tiny effective quota is just proportional scaling doing its job — see the note at the
bottom, it is not the bug.)
A client with `tenant_id="probe2"` then writes 256 × 1 MiB objects and reads back what is
present:
```
wrote_ok=256 rejected=0 of 256 MiB <- every put returned rc=0
still_present=5/256
after_45s_present=5/256 <- stable, not a transient
probe2: charged_bytes=5242880 over_quota=false admission_closed=false
mooncake_tenant_evict_bytes_total{tenant_id="probe2"} 275775488
```
251 of 256 objects were accepted and thrown away. `charged_bytes` sits exactly at the quota, so
accounting is correct — the tenant is simply being trimmed to its share on every write.
### Three separate problems
**1. Writes report success for data that is immediately discarded.**
Per RFC #2153 this should be `TENANT_QUOTA_EXCEEDED` once eviction cannot make room, but
eviction can always make room by evicting what was just written. A caller has no way to
distinguish "stored" from "accepted and dropped", so there is no signal to back off, shrink the
working set, or alert. For a cache this is a survivable semantic, but it should be an explicit
one rather than an accident of eviction ordering.
RFC #2153 leaves this as an open question:
> Should over-quota eviction run synchronously on mount/unmount, asynchronously, or only on
> future writes?
Our request is narrower: whatever the eviction policy, there should be a way to make an
over-quota write **fail** instead of silently evicting. A flag would be enough
(`-tenant_quota_reject_on_exceed`), or a rule that eviction may not select objects written by
the request currently being admitted.
**2. `mooncake_tenant_quota_reject_total` is declared but never exported.**
RFC #2153 specifies `mooncake_tenant_quota_reject_total{tenant_id,reason}`. The string is
present in the binary, but the metric does not appear in `/metrics` at all — before, during, or
after the run above. Anyone building an alert on it gets permanent silence. (This may simply
follow from problem 1: if the counter is only registered on first increment and rejection never
happens, it can never appear. Either way it is not usable as an alarm.)
**3. Tenant-quota eviction is invisible in the global eviction counters.**
Throughout the 263 MiB evicted above:
```
master_evicted_key_count 0
master_evicted_size_bytes 0
master_successful_evictions_total 0
```
Only `mooncake_tenant_evict_bytes_total` moved. A dashboard watching the documented global
eviction metrics shows a completely idle cluster while a tenant loses 98% of its writes. These
are real evictions and we would expect them to be counted as such, or for the relationship
between the two counter families to be documented.
### Environment
- `docker.io/kvcacheai/mooncake:0.3.13`, Kubernetes, 3 store nodes × 200 GiB, RDMA
- Master: `-enable_multi_tenants=true -tenant_quota_connector_type=etcd
-tenant_quota_connector_uri=:2379 -eviction_high_watermark_ratio=0.9 -eviction_ratio=0.05`
- Client: `MooncakeDistributedStore.setup(..., tenant_id="probe2")`
### Expected
At least one of:
- an over-quota write fails with `TENANT_QUOTA_EXCEEDED`, as RFC #2153 describes; or
- it is documented that over-quota is enforced by eviction and never by rejection, with
`mooncake_tenant_quota_reject_total` removed rather than left dead, and tenant eviction
reflected in the global eviction counters so it is at least observable.
### Aside, not the bug
The 5.9 MB effective quota above comes from proportional scaling working as designed:
`10 GiB / (1024 TiB + 10 GiB) × 600 GiB`. Setting one tenant to a very large requested value —
the natural way to express "this tenant gets the whole pool, whatever size it is today" — makes
every subsequently added tenant a rounding error. That is arguably correct arithmetic, but it is
a sharp edge, and it interacts badly with problem 1: the newly added tenant is starved *and*
silent. A warning when a tenant's effective quota falls below some fraction of its requested
value would have saved us an afternoon.
### Before submitting a new issue...
- [x] Searched existing issues and read the documentation. #2153 (RFC, closed) and #2612 (the
implementing PR, closed) are the design and implementation this report is measured
against; #3157 is a separate refactor of quota charge/release.
Contributor guide
Research direction
Start with RFC #2153 and implementing PR #2612, then trace the tenant admission and eviction path described in this report. Reproduce the probe2 workload and inspect /metrics, tenant eviction counters, global eviction counters, and quota status; done means the chosen over-quota behavior is explicit and the relevant rejection and eviction metrics are usable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, kubernetes
- Domain
- distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100