apache / apache/incubator-pegasus

How about introducing Rcoksdb BlobDB feature?

Open
#2,201 3 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
C++
Stars
2.1k
Forks
328
PR merge metrics
No merged PRs in 30d

Description

## Feature Request

**Is your feature request related to a problem? Please describe:**
We have some business requirements to write relatively large kv into the Pegasus.
For example, a user wants to write hundreds of thousands of data per second, and every singal date is about 80KB.
We have made many attempts to optimize the write parameters, such as adjust the memtable number/size of rocksdb,but they are far away from our target.
I noticed a paper named **WiscKey** https://www.usenix.org/system/files/conference/fast16/fast16-papers-lu.pdf, this paper proposes a solution to separate the key and value of RocksDB, which helps to significantly reduce the write amplification factor.
I read the logic of Rocksdb **BlobDB** carefully (Inspiration comes from **WiscKey**):https://rocksdb.org/blog/2021/05/26/integrated-blob-db.html, I think it's the time to introduce the feature to pegasus now.

**Describe the feature you'd like:**
I'd like to implement this feature in Pegasus.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reading the linked WiscKey paper and RocksDB BlobDB documentation, then inspect Pegasus's RocksDB integration to determine the design and compatibility boundaries. Done should include an agreed implementation scope and measurable acceptance criteria for large-value write performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.