kvcache-ai / kvcache-ai/Mooncake
[Bug]: File-per-key ScanMeta double-counts rescans and aborts on malformed files
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
## Bug Report
On current `main`, `StorageBackendAdaptor::ScanMeta()` is not safe as a recovery operation.
Two related problems are visible in the current implementation:
1. Every scan increments the member counters `total_keys` and `total_size` without resetting or recomputing them. `FileStorage::ReRegisterOffloadedObjects()` reruns `ScanMeta()` on the same backend after master metadata loss, so repeated recovery scans inflate the usage counters.
2. Each regular file is passed directly to `struct_pb::from_pb()`. The decoder throws on malformed/truncated input, but `ScanMeta()` does not catch per-record exceptions. A single partial or corrupted cache file can therefore escape the expected-returning startup or repeated master-recovery flow as an exception, instead of producing a recoverable scan error.
The existing adaptor test avoids calling `ScanMeta()` twice on one instance specifically to avoid double-counting, which leaves the production rescan path uncovered.
## Expected behavior
A successful full scan should atomically replace recovered usage totals rather than accumulate them. Any traversal, inspection, read, decode, or handler failure must fail closed: return an `ErrorCode`, preserve the last committed totals, leave `meta_scanned_` false, and avoid committing a partial local snapshot. Malformed or empty object files must not be silently skipped, because that would make logical accounting lower than physical disk occupancy. Once the underlying condition is repaired, a complete retry can re-register all objects and publish the new totals.
This follows the fail-closed recovery rule in #3034.
### Before submitting...
- [x] Searched existing issues and open PRs for rescan accounting and malformed file handling.
Contributor guide
Research direction
Start with StorageBackendAdaptor::ScanMeta() and trace FileStorage::ReRegisterOffloadedObjects() to understand the repeated recovery scan. Review the existing adaptor test, then add coverage for rescans and malformed or empty files. Done means a complete scan replaces totals atomically, while any traversal, read, decode, or handler failure preserves the last committed totals and leaves meta_scanned_ false.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100