kvcache-ai / kvcache-ai/Mooncake
[Bug]: PutEnd silently discards PushOffloadingQueue failures — objects never reach the SSD tier, with no log or metric
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
## Summary
With eager offload (`enable_offload=1`, `offload_on_evict=0`), `MasterService::PutEnd` calls `PushOffloadingQueue` and acts **only on success**. There is no `else` branch, no log, no metric, and no retry:
```cpp
// mooncake-store/src/master_service.cpp (main @ fc04cb1, ~L3283)
auto result = PushOffloadingQueue(object_id, replica);
if (result) {
if (!task_created) {
replica.inc_refcnt();
tenant_state.offloading_tasks.emplace(
object_id.user_key,
OffloadingTask{replica.id(), std::chrono::system_clock::now()});
task_created = true;
}
}
// failure is discarded here
```
An object whose enqueue fails therefore gets **no offload task**, is never written to the SSD tier, and dies RAM-only at the next memory eviction. Because it never had a disk replica, that eviction removes its last replica and the object leaves the pool entirely — a cache entry silently lost despite the SSD tier having free space.
## Failure modes that can hit this
`PushOffloadingQueue` (`master_service.cpp`, ~L5150) can return:
- `SEGMENT_NOT_FOUND` — segment name not present in the local-disk `client_by_name` map
- `UNABLE_OFFLOADING` — no local disk segment for the resolved client
- `UNABLE_OFFLOADING` — `!enable_offloading` on that segment
- `KEYS_ULTRA_LIMIT` — `offloading_objects.size() >= offloading_queue_limit_`
- `OBJECT_ALREADY_EXISTS` — key already present in `offloading_objects`
None of these are logged inside `PushOffloadingQueue` either, so the failure is invisible end-to-end.
## Why this is inconsistent
The **eviction-path** callers of the same function already count and log these failures, e.g.:
```cpp
LOG(WARNING) << "[EVICT] PushOffloadingQueue failed for "
<< offload_push_failed_forced
<< " object(s); force-evicted without disk offload "
"(offload_force_evict=true).";
```
Only the `PutEnd` hot-path caller discards them. There is also no offload-related metric anywhere in `master_metric_manager.{h,cpp}`.
## Impact observed
In a production deployment we found roughly **9% of pool exits had no corresponding disk eviction** — i.e. that fraction of objects left the cache without ever reaching SSD — while the SSD tier was well under its eviction watermark and the offload pipeline was healthy (multi-GB/s end-to-end store throughput, no queue backlog, no dropped events). With no log line and no counter anywhere, there was nothing to attribute it to; localising it required reading the source rather than any telemetry.
The measurement that surfaced it, for anyone hitting the same thing: compare cumulative pool exits against cumulative disk evictions. A persistent excess of the former means objects are leaving the cache without a disk replica.
Note this is distinct from #2632 / #2658 (offloading-task orphans causing 600s expiry warnings). That one produces a *visible* `Offloading task expired` warning. This path produces nothing at all.
## Two additional silent-success paths
`PushOffloadingQueue` also returns success (`{}`) while enqueuing nothing:
1. `if (segment_names.empty()) { return {}; }` — early return before the loop
2. a `segment_names` list whose entries are all `nullopt` falls through the loop body
In both cases `PutEnd` sees success and records an `OffloadingTask` + `inc_refcnt()` for work that was never queued, so the task can only be cleared by the TTL expiry path.
## Suggested fix
At minimum, make the failure observable: add an `else` incrementing an offload-enqueue-failure counter and emitting a (sampled, since this is the `PutEnd` hot path) `WARNING` carrying the `ErrorCode`, mirroring what the eviction-path callers already do. That is enough to tell operators which failure mode dominates.
Beyond that, `KEYS_ULTRA_LIMIT` and `OBJECT_ALREADY_EXISTS` arguably warrant retry or dedup handling rather than a dropped object, but making it visible seems like the necessary first step.
Happy to open a PR for the observability part if that is a welcome direction.
Environment: observed on `v0.3.12-pre1`; code path verified unchanged on `main` @ `fc04cb1`.
Contributor guide
Research direction
Start in mooncake-store/src/master_service.cpp at MasterService::PutEnd and PushOffloadingQueue, then compare the eviction-path failure handling described in the issue. Read master_metric_manager.{h,cpp} for existing metric patterns; done means PutEnd enqueue failures expose their ErrorCode through the requested counter and sampled warning without silently discarding the result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100