bitshares / bitshares/bitshares-core

Syncing issue around block 60444406

Open
#2,497 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.2k
Forks
660
Avg merge
8h 17m
Merged PRs (30d)
26

Description

**Bug Description**

There was a syncing issue around block 60444406.

The witness who generated this block was out of sync:
```
2021-07-24T22:24:06 th_a:invoke handle_block handle_block ] Got block: #60444399 039a4eef77b0463c94c13b135c442a5d539b83b5 time: 2021-07-24T22:24:06 transaction(s): 4 latency: 1
90 ms from: abc123 irreversible: 60444383 (-16) application.cpp:598
2021-07-24T22:24:07 th_a:invoke handle_transaction handle_transaction ] Got 2 transactions from network application.cpp:660
2021-07-24T22:24:08 th_a:invoke handle_transaction handle_transaction ] Got 3 transactions from network application.cpp:660
2021-07-24T22:24:12 th_a:Witness Block Production block_production_loo ] Generated block #60444400 with 6 transaction(s) and timestamp 2021-07-24T22:24:12 at time 2021-07-24T22:24:12
witness.cpp:272
2021-07-24T22:24:31 th_a:invoke handle_block _push_block ] Switching to fork: 039a4ef1bfec83cd809b767eee0a36087cfab8ee db_block.cpp:153
2021-07-24T22:24:31 th_a:invoke handle_block _push_block ] popping block #60444400 039a4ef0944c7dc718c5a767b94d24e47718be4f db_block.cpp:159
2021-07-24T22:24:31 th_a:invoke handle_block _push_block ] pushing block from fork #60444400 039a4ef0fd6db95988e7c1dc0fc63e30752094a9 db_block.cpp:166
2021-07-24T22:24:31 th_a:invoke handle_block _push_block ] pushing block from fork #60444401 039a4ef1bfec83cd809b767eee0a36087cfab8ee db_block.cpp:166
2021-07-24T22:24:32 th_a:Witness Block Production block_production_loo ] Generated block #60444406 with 0 transaction(s) and timestamp 2021-07-24T22:24:30 at time 2021-07-24T22:24:32
witness.cpp:272
2021-07-24T22:24:32 th_a:invoke handle_transaction handle_transaction ] Got 2 transactions from network application.cpp:660
2021-07-24T22:24:33 th_a:invoke handle_block handle_block ] Got block: #60444407 039a4ef70d36957be2cb88d13f11972843bb977d time: 2021-07-24T22:24:33 transaction(s): 7 latency: 4
98 ms from: delegate.freedom irreversible: 60444392 (-15) application.cpp:598
```

Most of other nodes got this block as a "sync block" so it did not show in `default.log`:
```
2021-07-24T22:24:24 th_a:invoke handle_block handle_block ] Got block: #60444404 039a4ef408560fc22579dd03962eff0783f5c362 time: 2021-07-24T22:24:24 transaction(s): 5 latency: 747 ms from: btspp-witness irreversible: 60444389 (-15) application.cpp:598
2021-07-24T22:24:26 th_a:invoke handle_transaction handle_transaction ] Got 4 transactions from network application.cpp:660
2021-07-24T22:24:27 th_a:invoke handle_transaction handle_transaction ] Got 5 transactions from network application.cpp:660
2021-07-24T22:24:27 th_a:invoke handle_block handle_block ] Got block: #60444405 039a4ef513a98e67ff51be747c4983e71e2be1df time: 2021-07-24T22:24:27 transaction(s): 6 latency: 670 ms from: magicwallet.witness irreversible: 60444390 (-15) application.cpp:598
2021-07-24T22:24:29 th_a:invoke handle_transaction handle_transaction ] Got 2 transactions from network application.cpp:660
2021-07-24T22:24:31 th_a:invoke handle_transaction handle_transaction ] Got 2 transactions from network application.cpp:660
2021-07-24T22:24:33 th_a:invoke handle_block handle_block ] Got block: #60444407 039a4ef70d36957be2cb88d13f11972843bb977d time: 2021-07-24T22:24:33 transaction(s): 7 latency: 189 ms from: delegate.freedom irreversible: 60444392 (-15) application.cpp:598
2021-07-24T22:24:34 th_a:invoke handle_transaction handle_transaction ] Got 1 transactions from network application.cpp:660
2021-07-24T22:24:36 th_a:invoke handle_block handle_block ] Got block: #60444408 039a4ef8553c978cce656047379ec3749ee1dfd9 time: 2021-07-24T22:24:36 transaction(s): 3 latency: 413 ms from: gbac-ety001 irreversible: 60444393 (-15) application.cpp:598
```

A few nodes got stuck:
![image](https://user-images.githubusercontent.com/9946777/126913816-57625c74-ba69-4eb3-b3d5-c7ebba5942dd.png)
![image](https://user-images.githubusercontent.com/9946777/126913921-3a54f6f0-ba6d-4b77-b871-0919e36344f7.png)
```
2021-07-24T22:24:27 th_a:invoke handle_block handle_block ] Got block: #60444405 039a4ef513a98e67ff51be747c4983e71e2be1df time: 2021-07-24T22:24:27 transaction(s): 6 latency: 232 ms from: magicwallet.witness irreversible: 6044
4390 (-15) application.cpp:546
2021-07-24T22:24:28 th_a:invoke handle_transaction handle_transaction ] Got 3 transactions from network application.cpp:615
2021-07-24T22:24:29 th_a:invoke handle_transaction handle_transaction ] Got 1 transactions from network application.cpp:615
2021-07-24T22:24:31 th_a:invoke handle_transaction handle_transaction ] Got 2 transactions from network application.cpp:615
2021-07-24T22:25:27 th_a:Witness Block Production block_production_loo ] Generated block #60444406 with 7 transaction(s) and timestamp 2021-07-24T22:25:27 at time 2021-07-24T22:25:27 witness.cpp:277
```

Getting stuck is the issue. Ideally those nodes should recover automatically by themselves. But they didn't.

Update: another instance
![image](https://user-images.githubusercontent.com/9946777/128402631-8f7e4a35-1717-4ec9-bb81-c7de3fd53f40.png)

**Impacts**
Describe which portion(s) of BitShares Core may be impacted by this bug. Please tick at least one box.
- [ ] API (the application programming interface)
- [ ] Build (the build process or something prior to compiled code)
- [ ] CLI (the command line wallet)
- [ ] Deployment (the deployment process after building such as Docker, Travis, etc.)
- [ ] DEX (the Decentralized EXchange, market engine, etc.)
- [ ] P2P (the peer-to-peer network for transaction/block propagation)
- [ ] Performance (system or user efficiency, etc.)
- [ ] Protocol (the blockchain logic, consensus, validation, etc.)
- [x] Security (the security of system or user data, etc.)
- [ ] UX (the User Experience)
- [ ] Other (please add below)

## CORE TEAM TASK LIST
- [ ] Evaluate / Prioritize Bug Report
- [ ] Refine User Stories / Requirements
- [ ] Define Test Cases
- [ ] Design / Develop Solution
- [ ] Perform QA/Testing
- [ ] Update Documentation

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the block and fork-handling paths referenced in application.cpp, witness.cpp, and db_block.cpp, using the supplied logs to compare normal sync blocks with nodes that become stuck. Reproduce or instrument the block-sync recovery flow, then verify that a node recovers automatically from the reported fork and continues processing later blocks.

Written by the indexing model from the issue text.

Assessment

Tech stack
blockchain, cpp
Domain
distributed-systems, networking, security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.