mafintosh / mafintosh/hyperdb

Writing large quantities of data into Hyperdb can be incredibly slow

Open
#74 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
748
Forks
56
PR merge metrics
No merged PRs in 30d

Description

I have not looked at this since v3.0.0-0 was released, but i dont think this issue has been resolved. Looking at the code now - it looks like the small cache that was present has been removed.

As discussed on gitter last month:

> I did a bit of profiling to see where the bottle neck is in importing data into hyperdb - it looks like its raf.read().
>
> ```
> ticks parent name
> 49282 41.9% node::Read(v8::FunctionCallbackInfo const&)
> 49282 100.0% v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*)
> 49257 99.9% LazyCompile: *fs.read fs.js:645:19
> 49257 100.0% LazyCompile: *RandomAccessFile._read /Users/x/hyper-graph-cli/node_modules/random-access-file/index.js:92:45
> 48824 99.1% LazyCompile: *Storage.getNode /Users/x/hyper-graph-cli/node_modules/hypercore/lib/storage.js:108:38
> 48466 99.3% LazyCompile: *Storage.dataOffset /Users/x/hyper-graph-cli/node_modules/hypercore/lib/storage.js:70:41
> ```
>
> After further analysis it looks like its caused by a cache overflow issue, but I think an unavoidable one unless you store all of hyperdb in memory which is probably not a good idea or do something super clever. The hyperdb writer already has a cache - https://github.com/mafintosh/hyperdb/blob/master/lib/writer.js#L25-L26. But when putting in new entries it looks like it needs to get information about the previous nodes to construct the trie. On a large db this quickly results in evictions from the cache as each new key has to get different nodes for the hypercore.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by profiling large writes around raf.read(), then inspect hyperdb/lib/writer.js and the cache behavior described in the issue. Compare cache evictions and write performance on a large database; done would require addressing the reported slowdown without assuming the entire database can remain in memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.