hyperledger / hyperledger/fabric

Orderer panics after being restored

Open
#3,223 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
16.7k
Forks
9.1k
Avg merge
5h 38m
Merged PRs (30d)
26

Description

Let's say we have three orderers on a channel, and they have been backed up at different heights:
Orderer1 at height 7
Orderer2 at height 8
Orderer3 at height 9

Restore Orderer1 and Orderer2 first.

Orderer2 is leader now:
2022-02-09 13:02:26.183 EST 0118 INFO [orderer.consensus.etcdraft] run -> Raft leader changed: 0 -> 2 channel=mychannel node=1

Orderer1 will get block 8. So far so good.

Then when you bring up Orderer3 there won't be a leader re-election, Orderer3 at height 9 will simply join the existing quorum.

Orderer3 says Orderer2 is still leader. Wait a few minutes to ensure everything is stable and then send a transaction.

Orderer1 and Orderer2 will create block 9.

Orderer3 will complain:
2022-02-09 13:27:54.043 EST 03c6 INFO [orderer.consensus.etcdraft] writeBlock -> Got block [9], expect block [10], this node was forced to catch up channel=mychannel node=3

Send another transaction, then Orderer3 will panic:

2022-02-09 13:28:57.997 EST 0466 PANI [[orderer.commmon.multichannel](http://orderer.commmon.multichannel/)] commitBlock -> [channel: mychannel] Could not append block: unexpected Previous block hash.

I'm posting the logs after the start of Orderer1, Orderer2, and eventually Orderer3 to demonstrate a new leader is not elected at height 9.
[orderer_logs.zip](https://github.com/hyperledger/fabric/files/8035584/orderer_logs.zip)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.