Writing large quantities of data into Hyperdb can be incredibly slow
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 748
- Forks
- 56
- PR merge metrics
- No merged PRs in 30d
Description
I have not looked at this since v3.0.0-0 was released, but i dont think this issue has been resolved. Looking at the code now - it looks like the small cache that was present has been removed.
As discussed on gitter last month:
> I did a bit of profiling to see where the bottle neck is in importing data into hyperdb - it looks like its raf.read().
>
> ```
> ticks parent name
> 49282 41.9% node::Read(v8::FunctionCallbackInfo const&)
> 49282 100.0% v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*)
> 49257 99.9% LazyCompile: *fs.read fs.js:645:19
> 49257 100.0% LazyCompile: *RandomAccessFile._read /Users/x/hyper-graph-cli/node_modules/random-access-file/index.js:92:45
> 48824 99.1% LazyCompile: *Storage.getNode /Users/x/hyper-graph-cli/node_modules/hypercore/lib/storage.js:108:38
> 48466 99.3% LazyCompile: *Storage.dataOffset /Users/x/hyper-graph-cli/node_modules/hypercore/lib/storage.js:70:41
> ```
>
> After further analysis it looks like its caused by a cache overflow issue, but I think an unavoidable one unless you store all of hyperdb in memory which is probably not a good idea or do something super clever. The hyperdb writer already has a cache - https://github.com/mafintosh/hyperdb/blob/master/lib/writer.js#L25-L26. But when putting in new entries it looks like it needs to get information about the previous nodes to construct the trie. On a large db this quickly results in evictions from the cache as each new key has to get different nodes for the hypercore.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by profiling large writes around raf.read(), then inspect hyperdb/lib/writer.js and the cache behavior described in the issue. Compare cache evictions and write performance on a large database; done would require addressing the reported slowdown without assuming the entire database can remain in memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100