tarantool / tarantool/tarantool

Make appliers deliver full raft state, like relays

Open
#10,039 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug raft
Dominant language
Lua
Stars
3.7k
Forks
419
Avg merge
1d 23h
Merged PRs (30d)
88

Description

Bug description
Nodes configured with election_mode='candidate' vote for themselves when they see a new term being written with no corresponding vote for someone. It's possible that some node starts an election, writes term and vote for self, but due to applier being faster than relay on this node, it first delivers a term without vote through applier, and only then sends a term with vote through relay. Some remote candidate might schedule a vote for self instead of voting for the election starter in this case.

We've already fixed a similar issue for box.ctl.promote(): https://github.com/tarantool/tarantool/issues/8497. There term and vote were delivered separately even through relay.

Now the same issue may occur when delivering a term through applier (which was added in https://github.com/tarantool/tarantool/commit/544955109140e5dd5b956178fe47fd5ed3da2952).

Note that even though the normal relay channel delivers term and vote atomically (RAFT: message {term: 3, vote: 3, state: candidate, vclock: {1: 23}} from 3), it's already too late because a term was delivered alone via the applier channel (RAFT: received a newer term from 3)

[010] storage3 | 2024-05-22 10:24:55.731 [28221] main/126/main I> RAFT: bump term to 3, follow
[010] storage3 | 2024-05-22 10:24:55.731 [28221] main/126/main I> RAFT: vote for 3, follow
[010] storage3 | 2024-05-22 10:24:55.731 [28221] main/120/raft_worker I> RAFT: persisted state {term: 3, vote: 3}
[010] storage3 | 2024-05-22 10:24:55.731 [28221] main/120/raft_worker I> RAFT: enter candidate state with 1 self vote
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/125/main I> RAFT: received a newer term from 3
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/125/main I> RAFT: bump term to 3, follow
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/119/raft_worker I> RAFT: persisted state {term: 3}
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/119/raft_worker I> RAFT: vote for 2, follow
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/119/raft_worker I> RAFT: persisted state {term: 3, vote: 2}
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/119/raft_worker I> RAFT: enter candidate state with 1 self vote
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/114/applier/replicator@unix/:./storage3.iproto I> RAFT: message {term: 3, vote: 3, state: candidate, vclock: {1: 23}} from 3
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/114/applier/replicator@unix/:./storage3.iproto I> RAFT: vote request is skipped - competing candidate

What should be done
We need to think this through more thoroughly. There are various possible fixes. We can stop taking such term bumps in ACKs into account when deciding to start new elections, or maybe we can rethink raft communications between nodes, for example, make appliers send full raft state in ACKs or maybe completely detach raft communications from the data stream.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how the applier and relay deliver Raft term and vote state, then compare that behavior with the earlier box.ctl.promote fix in issue 8497 and commit 544955109140e5dd5b956178fe47fd5ed3da2952. The issue leaves several fixes open; done requires choosing and validating a design that prevents a term from being delivered without its corresponding vote.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.