cayleygraph / cayleygraph/cayley

bolt: Performance degradation while loading Freebase dump

Open
#815 19 comments 2 reactions 1 assignee Claimed by @dennwc View on GitHub
bug performance
Dominant language
Go
Stars
15.1k
Forks
1.2k
PR merge metrics
No merged PRs in 30d

Description

**Description**
I'm trying to load full Freebase dump into cayley, but load performance is degrading, and it looks like the load process will never finish.

At first I've run a series of experiments to determine the fastest way to load the data. Here are the results (cells contain minutes needed to load the number of quads from the column header). As we can see `Bolt + pq + nosync + 2.5M` had the best performance (not sure though that nosync contributed at all)

  | 5M | 10M | 15M | 20M | 25M
-- | -- | -- | -- | -- | --
Bolt + nq + 10k | 2 | 7 | 16 | 20 | 29
Bolt + nq + 50k | 2 | 5 | 13 | 15 | 23
Bolt + nq + 100k | 2 | 5 | 12 | 14 | 20
Bolt + nq + 200k | 2 | 5 | 10 | 12 | 17
Bolt + nq + 500k | 2 | 5 | 8 | 10 | 13
Bolt + nq + 500k | 2 | 5 | 8 | 9 | 12
Bolt + nq + 1.25M | 2 | 5 | 7 | 9 | 12
Bolt + nq + 2.5M | 2 | 5 | 7 | 9 | 12
Bolt + nq + 5M | 3 | 5 | 8 | 9 | 12
Bolt + nq + 1M + nosync | 2 | 5 | 7 | 9 | 12
Bolt + nq + 2.5M + nosync | 2 | 5 | 7 | 9 | 11
Bolt + pq.gz + 1.25M | 2 | 4 | 7 | 8 | 10
Bolt + pq + nosync + 1.25M | 2 | 4 | 6 | 7 | 10
Bolt + pq + nosync + 2.5M | 2 | 4 | 6 | 7 | **9**
Leveldb + nq + buffer 20 + 10k | 4 | 12 | 26 |   |  
Leveldb + pq.gz + buffer 20M + 1.25M | 1 | 8 | 16 | 18 | 27
Leveldb + nq + buffer 20M + 5M | 2 | 18 |   |   |  
Leveldb + pq.gz + buffer 200M + 1.25M | 1 | 8 | 16 | 18 | 27
Leveldb + pq.gz + buffer 1G + 1.25M | 1 | 8 | 16 | 18 | 27
Leveldb + pq.gz + buffer 1G + 500k | 1 | 5 | 11 | 13 | 19
Leveldb + pq.gz + buffer 4G + 500k | 1 | 5 | 11 | 13 | 19
Leveldb + pq.gz + buffer 4G + 1.25M | 1 | 8 | 16 | 18 | 27
Leveldb + pq.gz + buffer 4G + cache 200M + 1.25M | 1 | 8 | 16 | 18 | 28

**Steps to reproduce the issue:**
1. `cayley load -c bolt.yml --verbose=3 -i freebase.pq`
2. bolt.yml:
```
store:
backend: bolt
address: bolt
load:
batch: 2500000
```
**Received results:**
At first, everything was ok, but then load started to slow down. The graph can demonstrate it better:
![image](https://user-images.githubusercontent.com/1741779/63259900-4fb34400-c288-11e9-8f25-5f8cf66b1fe5.png)

Then I've decided to make smaller batches and add `nosync`:
1. `cayley load -c bolt.yml --verbose=3 -i freebase.pq`
2. bolt.yml:
```
store:
backend: bolt
address: bolt
options:
nosync: true
load:
batch: 500000
```
Things became a bit better, but not for long:
![image](https://user-images.githubusercontent.com/1741779/63260040-9e60de00-c288-11e9-858b-8f4b4718fcc2.png)

`htop` says the process consumes more and more memory (58gb after 2 days)

**Expected results:**
Freebase loaded in less than infinity

**Output of `cayley version` or commit hash:**

```
Cayley version: 0.7.5
Git commit hash: cf576babb7db
```

**Environment details:**
CPU: 8 x Intel(R) Xeon(R) CPU @ 2.30GHz
Memory: 29Gb
OS: Ubuntu 16.04.6 LTS
Disk: SSD

Backend database: `(database and version)`
`Bolt`, not sure about version - cayley handled that for me

So the question is - am I doing it wrong or is there some bug?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.