cayleygraph / cayleygraph/cayley
bolt: Performance degradation while loading Freebase dump
- Dominant language
- Go
- Stars
- 15.1k
- Forks
- 1.2k
- PR merge metrics
- No merged PRs in 30d
Description
**Description**
I'm trying to load full Freebase dump into cayley, but load performance is degrading, and it looks like the load process will never finish.
At first I've run a series of experiments to determine the fastest way to load the data. Here are the results (cells contain minutes needed to load the number of quads from the column header). As we can see `Bolt + pq + nosync + 2.5M` had the best performance (not sure though that nosync contributed at all)
| 5M | 10M | 15M | 20M | 25M
-- | -- | -- | -- | -- | --
Bolt + nq + 10k | 2 | 7 | 16 | 20 | 29
Bolt + nq + 50k | 2 | 5 | 13 | 15 | 23
Bolt + nq + 100k | 2 | 5 | 12 | 14 | 20
Bolt + nq + 200k | 2 | 5 | 10 | 12 | 17
Bolt + nq + 500k | 2 | 5 | 8 | 10 | 13
Bolt + nq + 500k | 2 | 5 | 8 | 9 | 12
Bolt + nq + 1.25M | 2 | 5 | 7 | 9 | 12
Bolt + nq + 2.5M | 2 | 5 | 7 | 9 | 12
Bolt + nq + 5M | 3 | 5 | 8 | 9 | 12
Bolt + nq + 1M + nosync | 2 | 5 | 7 | 9 | 12
Bolt + nq + 2.5M + nosync | 2 | 5 | 7 | 9 | 11
Bolt + pq.gz + 1.25M | 2 | 4 | 7 | 8 | 10
Bolt + pq + nosync + 1.25M | 2 | 4 | 6 | 7 | 10
Bolt + pq + nosync + 2.5M | 2 | 4 | 6 | 7 | **9**
Leveldb + nq + buffer 20 + 10k | 4 | 12 | 26 | |
Leveldb + pq.gz + buffer 20M + 1.25M | 1 | 8 | 16 | 18 | 27
Leveldb + nq + buffer 20M + 5M | 2 | 18 | | |
Leveldb + pq.gz + buffer 200M + 1.25M | 1 | 8 | 16 | 18 | 27
Leveldb + pq.gz + buffer 1G + 1.25M | 1 | 8 | 16 | 18 | 27
Leveldb + pq.gz + buffer 1G + 500k | 1 | 5 | 11 | 13 | 19
Leveldb + pq.gz + buffer 4G + 500k | 1 | 5 | 11 | 13 | 19
Leveldb + pq.gz + buffer 4G + 1.25M | 1 | 8 | 16 | 18 | 27
Leveldb + pq.gz + buffer 4G + cache 200M + 1.25M | 1 | 8 | 16 | 18 | 28
**Steps to reproduce the issue:**
1. `cayley load -c bolt.yml --verbose=3 -i freebase.pq`
2. bolt.yml:
```
store:
backend: bolt
address: bolt
load:
batch: 2500000
```
**Received results:**
At first, everything was ok, but then load started to slow down. The graph can demonstrate it better:

Then I've decided to make smaller batches and add `nosync`:
1. `cayley load -c bolt.yml --verbose=3 -i freebase.pq`
2. bolt.yml:
```
store:
backend: bolt
address: bolt
options:
nosync: true
load:
batch: 500000
```
Things became a bit better, but not for long:

`htop` says the process consumes more and more memory (58gb after 2 days)
**Expected results:**
Freebase loaded in less than infinity
**Output of `cayley version` or commit hash:**
```
Cayley version: 0.7.5
Git commit hash: cf576babb7db
```
**Environment details:**
CPU: 8 x Intel(R) Xeon(R) CPU @ 2.30GHz
Memory: 29Gb
OS: Ubuntu 16.04.6 LTS
Disk: SSD
Backend database: `(database and version)`
`Bolt`, not sure about version - cayley handled that for me
So the question is - am I doing it wrong or is there some bug?
Contributor guide
Assessment
This issue has not been assessed yet.