Blockstream / Blockstream/esplora

[Electrs] history indexing is too slow, how to speed up?

Open
#220 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
1.3k
Forks
514
Avg merge
1d 10h
Merged PRs (30d)
16

Description

network: Bitcoin mainnet
daemon: local bitcoin full node
disk type: SSD 650G
RAM: 50G +
CPU: 24 cores

`txsore` indexing is very fast , each blk file only took few seconds, but `history` indexing is very slow, one blk file takes about 1 hour !

log:
```
DEBUG - writing 1921958 rows to RocksDB { path: "/database/btcmain/mainnet/newindex/history" }, flush=Disable
TRACE - parsing 134103055 bytes
TRACE - reading "/data/bitcoin/blocks/blk00018.dat"
TRACE - fetched 1786 blocks
DEBUG - writing 1942025 rows to RocksDB { path: "/database/btcmain/mainnet/newindex/history" }, flush=Disable
TRACE - parsing 134213401 bytes
TRACE - reading "/data/bitcoin/blocks/blk00019.dat"
TRACE - fetched 1856 blocks
DEBUG - writing 1954182 rows to RocksDB { path: "/database/btcmain/mainnet/newindex/history" }, flush=Disable
TRACE - parsing 134202443 bytes
TRACE - reading "/data/bitcoin/blocks/blk00020.dat"
TRACE - fetched 1384 blocks
DEBUG - writing 1965486 rows to RocksDB { path: "/database/btcmain/mainnet/newindex/history" }, flush=Disable
TRACE - parsing 134123997 bytes
TRACE - reading "/data/bitcoin/blocks/blk00021.dat"
TRACE - fetched 1566 blocks

```

![image](https://user-images.githubusercontent.com/30211020/86504570-5d17e500-bdec-11ea-8cfe-1a9619788f10.png)

`vdd` is my SSD,
```
[root@blockchain-node2 bin]# df -h /database/
Filesystem Size Used Avail Use% Mounted on
/dev/vdd1 700G 447G 254G 64% /database

```

```
[root@blockchain-node2 bin]# iostat -x
Linux 3.10.0-1062.9.1.el7.x86_64 (blockchain-node2) 07/04/2020 _x86_64_ (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
38.23 0.00 14.57 1.50 1.96 43.74

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.07 1.37 4.92 0.97 218.72 11.25 78.08 0.02 3.80 4.07 2.40 0.43 0.25
vdb 0.01 0.02 20.63 0.28 7918.99 74.88 764.29 0.09 4.58 4.59 3.49 0.83 1.74
vdc 0.01 0.11 74.80 1.38 2144.42 97.78 58.87 0.09 1.20 1.12 5.59 0.79 6.05
vdd 0.31 0.04 353.90 24.19 110664.79 12063.47 649.19 2.37 8.66 3.31 86.82 0.48 18.21
```

[root@blockchain-node2 bin]#

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.