kvcache-ai / kvcache-ai/Mooncake

[Bug]: File-per-key ScanMeta double-counts rescans and aborts on malformed files

Open
#2,982 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

## Bug Report

On current `main`, `StorageBackendAdaptor::ScanMeta()` is not safe as a recovery operation.

Two related problems are visible in the current implementation:

1. Every scan increments the member counters `total_keys` and `total_size` without resetting or recomputing them. `FileStorage::ReRegisterOffloadedObjects()` reruns `ScanMeta()` on the same backend after master metadata loss, so repeated recovery scans inflate the usage counters.
2. Each regular file is passed directly to `struct_pb::from_pb()`. The decoder throws on malformed/truncated input, but `ScanMeta()` does not catch per-record exceptions. A single partial or corrupted cache file can therefore escape the expected-returning startup or repeated master-recovery flow as an exception, instead of producing a recoverable scan error.

The existing adaptor test avoids calling `ScanMeta()` twice on one instance specifically to avoid double-counting, which leaves the production rescan path uncovered.

## Expected behavior

A successful full scan should atomically replace recovered usage totals rather than accumulate them. Any traversal, inspection, read, decode, or handler failure must fail closed: return an `ErrorCode`, preserve the last committed totals, leave `meta_scanned_` false, and avoid committing a partial local snapshot. Malformed or empty object files must not be silently skipped, because that would make logical accounting lower than physical disk occupancy. Once the underlying condition is repaired, a complete retry can re-register all objects and publish the new totals.

This follows the fail-closed recovery rule in #3034.

### Before submitting...

- [x] Searched existing issues and open PRs for rescan accounting and malformed file handling.

Contributor guide

Open the contributing guide

Research direction

Start with StorageBackendAdaptor::ScanMeta() and trace FileStorage::ReRegisterOffloadedObjects() to understand the repeated recovery scan. Review the existing adaptor test, then add coverage for rescans and malformed or empty files. Done means a complete scan replaces totals atomically, while any traversal, read, decode, or handler failure preserves the last committed totals and leaves meta_scanned_ false.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.