max-mapper / max-mapper/level-bulk-load
optimize write throughput!
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 11
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
hey all,
this repo contains a test file and some results based on some advice I got from @rescrv in ##leveldb IRC yesterday, the main idea being that in order to optimize throughput the size in bytes of a batch shouldnt exceed the writeBufferSize (default is 1MB), and that the sweet spot is around 16MB, with a maximum being 64MB (anything above 64 will be ignored)
i'm working with some large datasets and would really like to figure out the fastest way to insert huge amounts of data into leveldb.
my challenge: get a higher writes/sec for the 320,000 benchmark
with node v0.10.15 on an SSD (macbook air) it took 8m22s, or 637 ops/sec. this seems really low, surely someone can top it!
possible optimizations:
- different batching strategies. the non-batched version was only 130 ops/sec slower than the currently implemented batching version.
- different backends (hyperleveldb, basho leveldb are both game)
- micro optimizations in levelup/leveldown
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the repository's test file and the linked 320,000 benchmark, then compare the existing batching results with the proposed batch sizes and alternative backends. Done means demonstrating a higher writes-per-second result than the reported 637 ops/sec and documenting the approach used.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100