facebook / facebook/rocksdb

delta snapshots computation

Open
#12,298 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

I have a use case where i have a few large indexing machines that are ingesting all the data (in real time) and periodically (every X seconds) running a compaction and checkpointing the index to some distributed storage. The serving (read-only server) read these latest snapshots whenever new available (copy SST files from remote storage to local ssd). One limitation of this approach is if my indexes are pretty large for example few hundred GB to TB scale , taking snapshot of entire index , uploading them to remote storage , and the read-only servers loading these giant indexes every few minutes to the local SSD is taking a lot of bandwidth and affects performance on online path (even if the amount of updates in those N seconds was in order of few MBs) .

Is there a way to transfer only deltas between snapshots so the read servers can only load the new sst files and clean up the old files , without having to run any compaction on serving side and indexers instead of checkpointing the entire index , periodically updates the delta (with any metadata) ? Any ideas on how to accomplish this or if this is even feasible approach would be appreciated ?

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by surveying RocksDB's existing snapshot, checkpoint, and SST-file handling, then clarify the metadata and lifecycle requirements for incremental snapshots; the work is done when a concrete, maintainable approach is specified for transferring deltas and cleaning up serving-side files without serving-side compaction.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.