coinbase / coinbase/mesh-bitcoin

Mode to sync chain first + index later

Open
#52 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
119
Forks
127
PR merge metrics
No merged PRs in 30d

Description

I've been playing around with Rosetta for LBRY, and I've noticed I have lots of trouble on the initial sync + index when we start getting some larger blocks come in. I did see a recent discussion around improvements, but I haven't tried them yet (just saw them today).

Logs when this happens - almost looks like it fails to process a block in time, and then logs that block on shutdown along with the http request errors:

```
ss=0.005344 cache=2073.6MiB(5280827txo) IBD
2020-12-07T12:29:34Z UpdateTip: new best=2e51cbf8dfbd8820dd3bbdd55d1097c625414b7b8cb53190890eddfb135e1ab5 height=837104 version=0x20000000 log2_work=73.933872 txb=602 tx=20026401 date='2020-09-26T17:06:41Z' progress=0.005351 cache=2075.2MiB(5285550txo) IBD
2020-12-07T12:29:50Z UpdateTip: new best=6d6bfe8ca9db93626e88cc19e8ace45738730679995a0cf99bdba003b6cf19bb height=837117 version=0x20000000 log2_work=73.933907 txb=177 tx=20028012 date='2020-09-26T17:39:32Z' progress=0.005353 cache=2075.6MiB(5287003txo) IBD
2020-12-07T12:40:53Z tor: Thread interrupt
2020-12-07T12:40:53Z addcon thread exit
2020-12-07T12:40:53Z torcontrol thread exit
2020-12-07T12:40:53Z opencon thread exit
2020-12-07T12:40:53Z Shutdown: In progress...
2020-12-07T12:40:53Z net thread exit
2020-12-07T13:05:39Z UpdateTip: new best=7bfa4f19d91c3ed3dfa304ee3e82e2d7765bd8335d7802bcc24ad0e19ac02945 height=837138 version=0x20000000 log2_work=73.933963 txb=120 tx=20029990 date='2020-09-26T18:35:35Z' progress=0.005355 cache=57.7MiB(0txo) IBD
2020-12-07T13:05:39Z msghand thread exit
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z scheduler thread interrupt
2020-12-07T13:05:39Z Dumped mempool: 0.001684s to copy, 0.000714s to dump
2020-12-07T13:05:40Z Shutdown: done

```

Would it make sense to have a mode that lets the blockchain sync up fully before processing the blocks?

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the reported initial sync and indexing behavior and the recent discussion referenced in the issue. Reproduce the large-block sync scenario using the supplied logs as a guide, then define how a mode that syncs the chain before processing blocks should behave and how completion would be verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.