hashicorp / hashicorp/raft

If leader can't load snapshot cluster won't recover

Open
#522 3 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Go
Stars
9.1k
Forks
1.1k
Avg merge
3h 27m
Merged PRs (30d)
2

Description

Hello folks! I have a pretty lazy bug report here so apologies for not going deeper but I wanted to float a stance that by you and see if I can get away with it

We had a cluster of nomad servers that lost quorum and would not elect a new leader

Looking at the logs, the leader at the time was logging this

```
2022-08-17T03:11:20.634Z 2022-08-17T03:11:20.634Z [ERROR] snapshot: failed to get snapshots: error="open /run/nomad-server/server/raft/snapshots: no such file or directory"
2022-08-17T03:11:20.634Z 2022-08-17T03:11:20.634Z [ERROR] snapshot: failed to scan snapshot directory: error="open /run/nomad-server/server/raft/snapshots: no such file or directory"
2022-08-17T03:11:20.634Z failed to send snapshot to
2022-08-17T03:11:20.634Z failed to list snapshots
2022-08-17T03:11:20.634Z failed to get log
2022-08-17T03:11:20.608Z failed to list snapshots
2022-08-17T03:11:20.608Z failed to send snapshot to
```

And other servers were logging this
```
2022-08-17T03:07:55.677Z error waiting for Raft index error=timed out after 5s waiting for index=1525203
```

So here is my stance:
If the leader is broken because it cannot load the snapshots (I have no idea how we got in this situation but lets ignore that for now); the other server should realize the leader is useless and usurp him; perhaps via invoking the Praetorians Guard.

or more down to Earth: this state should cause a heartbeat failure in some way so that we can move past it and elect a new leader.

What do you think?

Contributor guide

Open the contributing guide

Research direction

Start by tracing the snapshot-loading failure from the reported missing snapshots directory into Raft heartbeat and leader-election handling. Reproduce the condition if possible using the logged errors, then determine whether a leader that cannot serve snapshots should trigger failure detection and a new election. Done means the failure mode is handled consistently and covered by an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.