apache / apache/incubator-pegasus
How about introducing Rcoksdb BlobDB feature?
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 328
- PR merge metrics
- No merged PRs in 30d
Description
## Feature Request
**Is your feature request related to a problem? Please describe:**
We have some business requirements to write relatively large kv into the Pegasus.
For example, a user wants to write hundreds of thousands of data per second, and every singal date is about 80KB.
We have made many attempts to optimize the write parameters, such as adjust the memtable number/size of rocksdb,but they are far away from our target.
I noticed a paper named **WiscKey** https://www.usenix.org/system/files/conference/fast16/fast16-papers-lu.pdf, this paper proposes a solution to separate the key and value of RocksDB, which helps to significantly reduce the write amplification factor.
I read the logic of Rocksdb **BlobDB** carefully (Inspiration comes from **WiscKey**):https://rocksdb.org/blog/2021/05/26/integrated-blob-db.html, I think it's the time to introduce the feature to pegasus now.
**Describe the feature you'd like:**
I'd like to implement this feature in Pegasus.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reading the linked WiscKey paper and RocksDB BlobDB documentation, then inspect Pegasus's RocksDB integration to determine the design and compatibility boundaries. Done should include an agreed implementation scope and measurable acceptance criteria for large-value write performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100