kvcache-ai / kvcache-ai/Mooncake

[Bug]: PutEnd silently discards PushOffloadingQueue failures — objects never reach the SSD tier, with no log or metric

Open
#2,997 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

## Summary

With eager offload (`enable_offload=1`, `offload_on_evict=0`), `MasterService::PutEnd` calls `PushOffloadingQueue` and acts **only on success**. There is no `else` branch, no log, no metric, and no retry:

```cpp
// mooncake-store/src/master_service.cpp (main @ fc04cb1, ~L3283)
auto result = PushOffloadingQueue(object_id, replica);
if (result) {
if (!task_created) {
replica.inc_refcnt();
tenant_state.offloading_tasks.emplace(
object_id.user_key,
OffloadingTask{replica.id(), std::chrono::system_clock::now()});
task_created = true;
}
}
// failure is discarded here
```

An object whose enqueue fails therefore gets **no offload task**, is never written to the SSD tier, and dies RAM-only at the next memory eviction. Because it never had a disk replica, that eviction removes its last replica and the object leaves the pool entirely — a cache entry silently lost despite the SSD tier having free space.

## Failure modes that can hit this

`PushOffloadingQueue` (`master_service.cpp`, ~L5150) can return:

- `SEGMENT_NOT_FOUND` — segment name not present in the local-disk `client_by_name` map
- `UNABLE_OFFLOADING` — no local disk segment for the resolved client
- `UNABLE_OFFLOADING` — `!enable_offloading` on that segment
- `KEYS_ULTRA_LIMIT` — `offloading_objects.size() >= offloading_queue_limit_`
- `OBJECT_ALREADY_EXISTS` — key already present in `offloading_objects`

None of these are logged inside `PushOffloadingQueue` either, so the failure is invisible end-to-end.

## Why this is inconsistent

The **eviction-path** callers of the same function already count and log these failures, e.g.:

```cpp
LOG(WARNING) << "[EVICT] PushOffloadingQueue failed for "
<< offload_push_failed_forced
<< " object(s); force-evicted without disk offload "
"(offload_force_evict=true).";
```

Only the `PutEnd` hot-path caller discards them. There is also no offload-related metric anywhere in `master_metric_manager.{h,cpp}`.

## Impact observed

In a production deployment we found roughly **9% of pool exits had no corresponding disk eviction** — i.e. that fraction of objects left the cache without ever reaching SSD — while the SSD tier was well under its eviction watermark and the offload pipeline was healthy (multi-GB/s end-to-end store throughput, no queue backlog, no dropped events). With no log line and no counter anywhere, there was nothing to attribute it to; localising it required reading the source rather than any telemetry.

The measurement that surfaced it, for anyone hitting the same thing: compare cumulative pool exits against cumulative disk evictions. A persistent excess of the former means objects are leaving the cache without a disk replica.

Note this is distinct from #2632 / #2658 (offloading-task orphans causing 600s expiry warnings). That one produces a *visible* `Offloading task expired` warning. This path produces nothing at all.

## Two additional silent-success paths

`PushOffloadingQueue` also returns success (`{}`) while enqueuing nothing:

1. `if (segment_names.empty()) { return {}; }` — early return before the loop
2. a `segment_names` list whose entries are all `nullopt` falls through the loop body

In both cases `PutEnd` sees success and records an `OffloadingTask` + `inc_refcnt()` for work that was never queued, so the task can only be cleared by the TTL expiry path.

## Suggested fix

At minimum, make the failure observable: add an `else` incrementing an offload-enqueue-failure counter and emitting a (sampled, since this is the `PutEnd` hot path) `WARNING` carrying the `ErrorCode`, mirroring what the eviction-path callers already do. That is enough to tell operators which failure mode dominates.

Beyond that, `KEYS_ULTRA_LIMIT` and `OBJECT_ALREADY_EXISTS` arguably warrant retry or dedup handling rather than a dropped object, but making it visible seems like the necessary first step.

Happy to open a PR for the observability part if that is a welcome direction.

Environment: observed on `v0.3.12-pre1`; code path verified unchanged on `main` @ `fc04cb1`.

Contributor guide

Open the contributing guide

Research direction

Start in mooncake-store/src/master_service.cpp at MasterService::PutEnd and PushOffloadingQueue, then compare the eviction-path failure handling described in the issue. Read master_metric_manager.{h,cpp} for existing metric patterns; done means PutEnd enqueue failures expose their ErrorCode through the requested counter and sampled warning without silently discarding the result.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.