Blockstream / Blockstream/esplora
[Electrs] history indexing is too slow, how to speed up?
- Dominant language
- JavaScript
- Stars
- 1.3k
- Forks
- 514
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 16
Description
network: Bitcoin mainnet
daemon: local bitcoin full node
disk type: SSD 650G
RAM: 50G +
CPU: 24 cores
`txsore` indexing is very fast , each blk file only took few seconds, but `history` indexing is very slow, one blk file takes about 1 hour !
log:
```
DEBUG - writing 1921958 rows to RocksDB { path: "/database/btcmain/mainnet/newindex/history" }, flush=Disable
TRACE - parsing 134103055 bytes
TRACE - reading "/data/bitcoin/blocks/blk00018.dat"
TRACE - fetched 1786 blocks
DEBUG - writing 1942025 rows to RocksDB { path: "/database/btcmain/mainnet/newindex/history" }, flush=Disable
TRACE - parsing 134213401 bytes
TRACE - reading "/data/bitcoin/blocks/blk00019.dat"
TRACE - fetched 1856 blocks
DEBUG - writing 1954182 rows to RocksDB { path: "/database/btcmain/mainnet/newindex/history" }, flush=Disable
TRACE - parsing 134202443 bytes
TRACE - reading "/data/bitcoin/blocks/blk00020.dat"
TRACE - fetched 1384 blocks
DEBUG - writing 1965486 rows to RocksDB { path: "/database/btcmain/mainnet/newindex/history" }, flush=Disable
TRACE - parsing 134123997 bytes
TRACE - reading "/data/bitcoin/blocks/blk00021.dat"
TRACE - fetched 1566 blocks
```

`vdd` is my SSD,
```
[root@blockchain-node2 bin]# df -h /database/
Filesystem Size Used Avail Use% Mounted on
/dev/vdd1 700G 447G 254G 64% /database
```
```
[root@blockchain-node2 bin]# iostat -x
Linux 3.10.0-1062.9.1.el7.x86_64 (blockchain-node2) 07/04/2020 _x86_64_ (24 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
38.23 0.00 14.57 1.50 1.96 43.74
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.07 1.37 4.92 0.97 218.72 11.25 78.08 0.02 3.80 4.07 2.40 0.43 0.25
vdb 0.01 0.02 20.63 0.28 7918.99 74.88 764.29 0.09 4.58 4.59 3.49 0.83 1.74
vdc 0.01 0.11 74.80 1.38 2144.42 97.78 58.87 0.09 1.20 1.12 5.59 0.79 6.05
vdd 0.31 0.04 353.90 24.19 110664.79 12063.47 649.19 2.37 8.66 3.31 86.82 0.48 18.21
```
[root@blockchain-node2 bin]#
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.