When the leader’s raft.LogStore hangs indefinitely, the network will hang indefinitely instead of re-electing

I recently encountered an issue in my production deployment of https://robustirc.net/, where the network was not making any Raft progress anymore.

It turns out that one of my servers has an issue with its (local NVMe) storage, which manifests itself in hanging indefinitely. There are no read or write errors, any disk access just hangs.

When that server happens to currently be the Raft leader when the issue occurs, **the entire Raft network will just hang indefinitely**. By this, I mean the leader will still participate in the Raft protocol (last contact times do update on the Raft followers), but applying new messages to Raft will not work (will timeout) and, crucially, the Raft network never even starts electing a new leader.

This was a surprising failure mode to me, and I wonder if that’s intentional (out of scope for Raft) or an issue with the current implementation?

To reproduce, I cloned the Raft example https://github.com/yongman/leto and modified it like so:

```diff
diff --git i/store/store.go w/store/store.go
index 6997242..2d63d77 100644
--- i/store/store.go
+++ w/store/store.go
@@ -9,10 +9,14 @@ package store
import (
"encoding/json"
"errors"
+ fmt "fmt"
"log"
"net"
"os"
+ "os/signal"
"path/filepath"
+ "sync"
+ "syscall"
"time"

"github.com/hashicorp/raft"
@@ -46,6 +50,51 @@ func NewStore(raftdir, raftbind string) (*Store, error) {
}, nil
}

+type hangingDB struct {
+ raft.LogStore
+
+ hangingMu sync.Mutex
+ hanging bool
+}
+
+func (h *hangingDB) currentlyHanging() bool {
+ h.hangingMu.Lock()
+ defer h.hangingMu.Unlock()
+ return h.hanging
+}
+
+func (h *hangingDB) hang() {
+ h.hangingMu.Lock()
+ defer h.hangingMu.Unlock()
+ h.hanging = true
+}
+
+func (h *hangingDB) StoreLog(log *raft.Log) error {
+ if h.currentlyHanging() {
+ time.Sleep(24 * time.Hour)
+ }
+ return h.LogStore.StoreLog(log)
+}
+
+func (h *hangingDB) StoreLogs(logs []*raft.Log) error {
+ if h.currentlyHanging() {
+ time.Sleep(24 * time.Hour)
+ }
+ return h.LogStore.StoreLogs(logs)
+}
+
+func newHangingDB(logStore raft.LogStore) *hangingDB {
+ h := &hangingDB{LogStore: logStore}
+ c := make(chan os.Signal, 1)
+ signal.Notify(c, syscall.SIGUSR1)
+ go func() {
+ sig := <-c
+ fmt.Printf("got signal: %v\n", sig)
+ h.hang()
+ }()
+ return h
+}
+
func (s *Store) Open(bootstrap bool, localID string) error {
config := raft.DefaultConfig()
config.LocalID = raft.ServerID(localID)
@@ -73,7 +122,7 @@ func (s *Store) Open(bootstrap bool, localID string) error {
}

// raft system
- r, err := raft.NewRaft(config, s.fsm, boltDB, boltDB, ss, transport)
+ r, err := raft.NewRaft(config, s.fsm, newHangingDB(boltDB), boltDB, ss, transport)
if err != nil {
return err
}

```

Then, after bringing up 3 nodes as described in the README, I send SIGUSR1 to the leader node, and now all commands hang, but no re-election happens.

The same is reproducible in https://robustirc.net/ with the robustirc-localnet command, but might be a bit more elaborate to test than the more self-contained leto example.

Given that it happens in two different projects built on top of hashicorp/raft, I don’t think it’s a bug with my code itself, but maybe both projects are using hashicorp/raft slightly wrong?

Any recommendations for how to handle this failure mode?

Start with the three-node Raft example and the modified store/store.go reproduction, then inspect how a leader's LogStore operations interact with election timeouts. Reproduce the SIGUSR1-triggered hang and compare leader-election and command behavior with a healthy node; done means the failure mode is either addressed or its intended handling is clearly established.

Written by the indexing model from the issue text.