If leader can't load snapshot cluster won't recover
- Dominant language
- Go
- Stars
- 9.1k
- Forks
- 1.1k
- Avg merge
- 3h 27m
- Merged PRs (30d)
- 2
Description
Hello folks! I have a pretty lazy bug report here so apologies for not going deeper but I wanted to float a stance that by you and see if I can get away with it
We had a cluster of nomad servers that lost quorum and would not elect a new leader
Looking at the logs, the leader at the time was logging this
```
2022-08-17T03:11:20.634Z 2022-08-17T03:11:20.634Z [ERROR] snapshot: failed to get snapshots: error="open /run/nomad-server/server/raft/snapshots: no such file or directory"
2022-08-17T03:11:20.634Z 2022-08-17T03:11:20.634Z [ERROR] snapshot: failed to scan snapshot directory: error="open /run/nomad-server/server/raft/snapshots: no such file or directory"
2022-08-17T03:11:20.634Z failed to send snapshot to
2022-08-17T03:11:20.634Z failed to list snapshots
2022-08-17T03:11:20.634Z failed to get log
2022-08-17T03:11:20.608Z failed to list snapshots
2022-08-17T03:11:20.608Z failed to send snapshot to
```
And other servers were logging this
```
2022-08-17T03:07:55.677Z error waiting for Raft index error=timed out after 5s waiting for index=1525203
```
So here is my stance:
If the leader is broken because it cannot load the snapshots (I have no idea how we got in this situation but lets ignore that for now); the other server should realize the leader is useless and usurp him; perhaps via invoking the Praetorians Guard.
or more down to Earth: this state should cause a heartbeat failure in some way so that we can move past it and elect a new leader.
What do you think?
Contributor guide
Research direction
Start by tracing the snapshot-loading failure from the reported missing snapshots directory into Raft heartbeat and leader-election handling. Reproduce the condition if possible using the logged errors, then determine whether a leader that cannot serve snapshots should trigger failure detection and a new election. Done means the failure mode is handled consistently and covered by an appropriate test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100