google / google/leveldb

Write Amplification factor for small key-values with frequent writes to levelDB

Open
#1,143 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
39.4k
Forks
8.2k
PR merge metrics
No merged PRs in 30d

Description

Hi Team,

I am facing a serious issue with write amplification. Consider, There are key-values that will write to levelDB for every 5 seconds. Those key-values are always batched into a write_batch object and then written to levelDB using leveldb_write() for every 5 sec. You can assume total size of key-values will be 500 bytes.

With leveldb ( sync = false ) , I am getting eMMC write count of 221 writes. where, If I write values directly into eMMC by maintaining some offsets for each value, I am getting eMMC write count of 70 writes.

The difference is Huge. Is there any way to optimize leveldb eMMC write count ? FYI, I tried all options available in Options.h file.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Options.h and the reported leveldb_write() and write_batch usage, then reproduce the five-second, 500-byte write workload with sync disabled. Compare LevelDB's eMMC write count with direct writes and determine whether a specific configuration or documented limitation explains the difference; done requires a validated optimization or a clearly supported conclusion.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.