apache / apache/kvrocks

Kvrocks & IngestExternalFile

Open
#2,458 1 comment 1 reaction 0 assignees View on GitHub
enhancement help wanted
Dominant language
C++
Stars
4.4k
Forks
658
Avg merge
1d 20h
Merged PRs (30d)
10

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.

### Motivation

Hi folks,

First of all - I just wanted to say that this is an awesome project 🙂

Secondly -

I wondered whether it's possible to load data to Kvrocks via RocksDB's [IngestExternalFile](https://github.com/facebook/rocksdb/wiki/creating-and-ingesting-sst-files).

The use case is real-world.

I currently work on a system that relies on (non-distributed) RocksDB, and we'd like to possible start using Kvrocks instead.
Every once in a while, we use an offline, "bulky" Spark process which essentially generates a complete view of the RocksDB database.
This is done by creating SST files directly, which is pretty cool*.
The system then downloads these files locally and just points RocksDB to use them.
This way, we can leverage Spark's super-scalable compute to create a dataset (of ~20B tiny records) which would otherwise take a long, long time to write to an empty RocksDB database.

**Q:**
Since Kvrocks uses RocksDB as its backend, I wondered - how hard would it be to do something like this?

Thanks!

---

* Technically: we create a Spark dataframe with the data in two columns, and we sort the data globally by key.
Then, a custom Spark module we've written creates an SST file using the RocksJava binding.
Each dataframe partition is turned into a separate SST file.
Since the data is sorted and keys are strictly unique, the SST files are non-overlapping, and thus are ingested to the bottomest level of RocksDB.

### Solution

I assume that a solution would involve the following components:

1. An offline library to create Kvrocks-compatible SSTs (i.e. conforming to [this](https://kvrocks.apache.org/community/data-structure-on-rocksdb))
2. A server API which can be given a list of SSTs to download and create a Kvrocks set from, using RockDB's IngestExternalFile.

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no repository files or tests. Start by reading RocksDB's IngestExternalFile documentation and Kvrocks' data-structure-on-rocksdb page, then inspect the server API entry points and storage integration needed for compatibility. Done would require an agreed design covering compatible SST creation and the API workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
redis
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.