hyperledger / hyperledger/fabric
Orderer panics after being restored
- Dominant language
- Go
- Stars
- 16.7k
- Forks
- 9.1k
- Avg merge
- 5h 38m
- Merged PRs (30d)
- 26
Description
Let's say we have three orderers on a channel, and they have been backed up at different heights:
Orderer1 at height 7
Orderer2 at height 8
Orderer3 at height 9
Restore Orderer1 and Orderer2 first.
Orderer2 is leader now:
2022-02-09 13:02:26.183 EST 0118 INFO [orderer.consensus.etcdraft] run -> Raft leader changed: 0 -> 2 channel=mychannel node=1
Orderer1 will get block 8. So far so good.
Then when you bring up Orderer3 there won't be a leader re-election, Orderer3 at height 9 will simply join the existing quorum.
Orderer3 says Orderer2 is still leader. Wait a few minutes to ensure everything is stable and then send a transaction.
Orderer1 and Orderer2 will create block 9.
Orderer3 will complain:
2022-02-09 13:27:54.043 EST 03c6 INFO [orderer.consensus.etcdraft] writeBlock -> Got block [9], expect block [10], this node was forced to catch up channel=mychannel node=3
Send another transaction, then Orderer3 will panic:
2022-02-09 13:28:57.997 EST 0466 PANI [[orderer.commmon.multichannel](http://orderer.commmon.multichannel/)] commitBlock -> [channel: mychannel] Could not append block: unexpected Previous block hash.
I'm posting the logs after the start of Orderer1, Orderer2, and eventually Orderer3 to demonstrate a new leader is not elected at height 9.
[orderer_logs.zip](https://github.com/hyperledger/fabric/files/8035584/orderer_logs.zip)
Contributor guide
Assessment
This issue has not been assessed yet.