facebook / facebook/rocksdb

Merge two rocksdb instances

Open
#5,582 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

We want to implement a merge operation of two rocksdb instance in our distributed table store like hbase.
Firstly, we try repairdb, but we found that repairdb need same cf name with same cf id. It's not a strongly guarantee in rocksdb implements.
Then, we try to use ingest file, but ingest file need has table properties ingest-file.version and ingest-file.global_seq which not exists in normal sst.
So we add ingest-file.version and ingest-file.global_seq in all normal files with version=2 and global_seq=-1 (let rocksdb read key's real seq. If we don't set it, it will read largest seq in manifest. If we set it to other value, it will let it to be global seq, not key's real seq).
But it must need a pwrite which distributed file system often not implements.
It also let rocksdb sst which be checkpointed (hard link) changed.
I think add a flag link ingest-file.version in manifest maybe make sense.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.