lbrycrdd block check performance degrades non-linearly with txset
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 173
- PR merge metrics
- No merged PRs in 30d
Description
**Background**
I run Luxor mining pool. Since ~5/21 we've seen larger numbers of transactions on the network, which is a good thing, but we've seen this impact the performance of lbrycrdd significantly.
**Issue**
Blocks should not take >1s to Connect. If they do, this causes a huge problem for miners, especially for short block times like Lbry.
**Expectation**
Performance in the same order of magnitude as BTC -
Bench output from lbrycrdd for block 771809
```
2020-05-28T16:11:32Z - Load block from disk: 18.95ms [2.92s]
2020-05-28T16:11:32Z - Sanity checks: 1.92ms [1.32s (0.88ms/blk)]
2020-05-28T16:11:32Z - Fork checks: 0.05ms [0.08s (0.05ms/blk)]
2020-05-28T16:11:34Z - Connect 398 transactions: 1822.49ms (4.579ms/tx, 2.925ms/txin) [327.93s (217.75ms/blk)]
2020-05-28T16:11:34Z - Verify 623 txins: 1822.81ms (2.926ms/txin) [328.06s (217.84ms/blk)]
2020-05-28T16:11:34Z - Index writing: 11.27ms [8.90s (5.91ms/blk)]
2020-05-28T16:11:34Z - Callbacks: 0.05ms [0.09s (0.06ms/blk)]
2020-05-28T16:11:34Z - Connect total: 1836.48ms [338.82s (224.98ms/blk)]
2020-05-28T16:11:34Z TrieCache size: 30670 nodes on block 771810, batch writes 3671856 bytes.
2020-05-28T16:11:34Z - Flush: 95.29ms [68.03s (45.17ms/blk)]
2020-05-28T16:11:34Z - Writing chainstate: 0.06ms [0.09s (0.06ms/blk)]
2020-05-28T16:11:34Z UpdateTip: new best=f46a34be4f5c791cf0b1d8667bae134c7eb10ee3dcfd89f04514d2e44a39cbdf height=771809 version=0x20000000 log2_work=73.716032 txb=398 tx=13122143 date='2020-05-28T15:47:20Z' progress=0.937661 cache=176.6MiB(548393txo)
2020-05-28T16:11:34Z - Connect postprocess: 0.82ms [2.73s (1.81ms/blk)]
2020-05-28T16:11:34Z - Connect block: 1951.61ms [412.60s (273.97ms/blk)]
```
Bench output for a similar block from btc (height 632057)
```
2020-05-28T16:32:32Z - Load block from disk: 26.86ms [12.35s]
2020-05-28T16:32:32Z - Sanity checks: 6.81ms [1.82s (2.91ms/blk)]
2020-05-28T16:32:32Z - Fork checks: 0.10ms [0.04s (0.06ms/blk)]
2020-05-28T16:32:33Z - Connect 3034 transactions: 517.85ms (0.171ms/tx, 0.091ms/txin) [466.69s (746.70ms/blk)]
2020-05-28T16:32:33Z - Verify 5688 txins: 518.47ms (0.091ms/txin) [471.33s (754.12ms/blk)]
2020-05-28T16:32:33Z - Index writing: 18.54ms [11.76s (18.82ms/blk)]
2020-05-28T16:32:33Z - Callbacks: 0.18ms [0.06s (0.10ms/blk)]
2020-05-28T16:32:33Z - Connect total: 545.06ms [485.48s (776.77ms/blk)]
2020-05-28T16:32:33Z - Flush: 14.22ms [8.43s (13.48ms/blk)]
2020-05-28T16:32:33Z - Writing chainstate: 0.08ms [0.05s (0.07ms/blk)]
2020-05-28T16:32:33Z UpdateTip: new best=00000000000000000009a30ace6dd38643db1798e34c15a8119e067ee0d4eed9 height=632057 version=0x20c00000 log2_work=91.984454 tx=533818995 date='2020-05-28T16:18:15Z' progress=0.999994 cache=384.0MiB(2932736txo) warning='74 of last 100 blocks have unexpected version'
2020-05-28T16:32:33Z - Connect postprocess: 6.06ms [7.94s (12.70ms/blk)]
2020-05-28T16:32:33Z - Connect block: 592.27ms [514.24s (822.78ms/blk)]
```
The most concerning problem is the tx validation speed. You'll notice it's 50x slower on LBRY than BTC.
Further, it seems to get worse as more transactions are added. Notice lbry block 771819 only has 55, and processes at 0.493 ms / txin.
```Verify 55 txins: 27.14ms (0.493ms/txin)```
This leads me to believe there is some non-linear regression
**Reproducer**
I can reliably reproduce this issue by running the lbrycrdd daemon with -debug=bench and see how long the block connect takes.
**Version:**
LBRYcrd Core Daemon version v0.17.3.2-be118de
**Machine**
2015 Macbook Pro, 2.8 GHz Quad-Core Intel Core i7
16 GB 1600 MHz DDR3
**Config**
Default config.
Contributor guide
Research direction
Reproduce the slowdown in the lbrycrdd daemon with -debug=bench, focusing on the cited block heights and the Connect and Verify timings. Compare the LBRY output with the BTC benchmark and the 55-input example to identify where validation becomes non-linear. Done means block connection is under one second and validation no longer degrades disproportionately as transactions increase.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- blockchain, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100