hashicorp / hashicorp/nomad

Candidate server failed to vote for itself (Can a server be a non-voter or is this a bug?)

Open
#13,889 7 comments 0 reactions 1 assignee Claimed by @tgross View on GitHub
theme/raft type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Looking for some guidance at least; I have logs from a server that indicates it never voted for itself. My question is, is this possible or a bug?

Details:
We restarted a bunch of servers in a datacenter and lost quorum. When we brought servers back, nomad failed to elect a leader. I am trying to understand why that happened.

In digging I found this strange behavior and need advice. Take term=252218 as an example

```
2022-07-14T03:54:59.965Z entering candidate state
2022-07-14T03:54:59.965Z votes needed=3
2022-07-14T03:54:59.966Z failed to make requestVote RPC dial tcp 10.131.8.33:4647: connect: connection refused
2022-07-14T03:55:01.489Z Election timeout reached, restarting election
```

This server entered a candidate state but never voted for itself, it never once prints this:
` r.logger.Debug("vote granted", "from", vote.voterID, "term", vote.Term, "tally", grantedVotes)`
How is this possible? The only thing that makes sense to me is perhaps this loop never contains itself; which means either 1. theres a bug where the server doesnt see itself in the server list or 2. the server isnt a voter
```
for _, server := range r.configurations.latest.Servers {
if server.Suffrage == Voter {
if server.ID == r.localID {
```

For the avoidance of doubt, this is not set and we do not have enterprise nomad:
https://www.nomadproject.io/docs/configuration/server#non_voting_server

Really appreciate guidance on how this is supposed to work to help me investigate. Digging through raft to find out where sufferage can change is a nightmare

### Nomad version
`Nomad v1.2.6`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.