hyperledger / hyperledger/fabric

Failed getting channel mychannel's ledger: error decoding the block number: error decoding varint with proto.Buffer: unexpected EOF

Open
#4,162 6 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
16.7k
Forks
9.1k
Avg merge
5h 38m
Merged PRs (30d)
26

Description

### Description

**Network configuration**
1. Fabric version: 2.4
2. Consensus: RAFT
3. Single Org setup with 2 peers and 3 orderers

**The above network was deployed using kubernetes

I was load testing the network with caliper where I created a scenario where disk got filled and for the first few times the orderers and peers where crashing afterwards they were able to restart when space was available but once in a while the orderers go into crash back loop off with the following error

2023-04-17 05:35:10.193 UTC 0004 INFO [orderer.common.server] initializeServerConfig -> Starting orderer with TLS enabled
2023-04-17 05:35:10.279 UTC 0005 INFO [orderer.common.server] Main -> Not bootstrapping the system channel because of existing channels
2023-04-17 05:35:10.288 UTC 0006 INFO [blkstorage] syncIndex -> Start building index from block [4144] to last block [5237]
2023-04-17 05:35:10.288 UTC 0007 PANI [orderer.common.server] extractSystemChannel -> Failed getting channel mychannel's ledger: error decoding the block number: error decoding varint with proto.Buffer: unexpected EOF
panic: Failed getting channel mychannel's ledger: error decoding the block number: error decoding varint with proto.Buffer: unexpected EOF

goroutine 1 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0000bc000, {0x0, 0x0, 0x0})
/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:234 +0x49b
go.uber.org/zap.(*SugaredLogger).log(0xc00059c148, 0x4, {0xf70ad2?, 0x0?}, {0xc000511290?, 0xc000511250?, 0x8698db?}, {0x0, 0x0, 0x0})
/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:234 +0x13b
go.uber.org/zap.(*SugaredLogger).Panicf(...)
/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:159
github.com/hyperledger/fabric/common/flogging.(*FabricLogger).Panicf(...)
/go/src/github.com/hyperledger/fabric/common/flogging/zap.go:74
github.com/hyperledger/fabric/orderer/common/server.extractSystemChannel({0x10cbe20, 0xc0001b2660}, {0x10d0c68, 0xc0005a88c0})
/go/src/github.com/hyperledger/fabric/orderer/common/server/main.go:406 +0x188
github.com/hyperledger/fabric/orderer/common/server.Main()
/go/src/github.com/hyperledger/fabric/orderer/common/server/main.go:149 +0x965
main.main()
/go/src/github.com/hyperledger/fabric/cmd/orderer/main.go:15 +0x17

### Steps to reproduce

1. Bring up HLF network with 1 Org which has 2 peers , 3 orderer with system-channel and application channel (mychannel)
2. Deploy Basic chaincode on the network
3. Create a file dump such that there is only few MB space left on the file system
4. Initiate load test using caliper
[orderer.log](https://github.com/hyperledger/fabric/files/11246617/orderer.log)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.