max-mapper / max-mapper/level-bulk-load

optimize write throughput!

Open
#1 17 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
11
Forks
2
PR merge metrics
No merged PRs in 30d

Description

hey all,

this repo contains a test file and some results based on some advice I got from @rescrv in ##leveldb IRC yesterday, the main idea being that in order to optimize throughput the size in bytes of a batch shouldnt exceed the writeBufferSize (default is 1MB), and that the sweet spot is around 16MB, with a maximum being 64MB (anything above 64 will be ignored)

i'm working with some large datasets and would really like to figure out the fastest way to insert huge amounts of data into leveldb.

my challenge: get a higher writes/sec for the 320,000 benchmark

with node v0.10.15 on an SSD (macbook air) it took 8m22s, or 637 ops/sec. this seems really low, surely someone can top it!

possible optimizations:

  • different batching strategies. the non-batched version was only 130 ops/sec slower than the currently implemented batching version.
  • different backends (hyperleveldb, basho leveldb are both game)
  • micro optimizations in levelup/leveldown

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the repository's test file and the linked 320,000 benchmark, then compare the existing batching results with the proposed batch sizes and alternative backends. Done means demonstrating a higher writes-per-second result than the reported 637 ops/sec and documenting the approach used.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.