tarantool / tarantool/tarantool
Make appliers deliver full raft state, like relays
Nobody has claimed this yet.
- Dominant language
- Lua
- Stars
- 3.7k
- Forks
- 419
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 88
Description
Bug description
Nodes configured with election_mode='candidate' vote for themselves when they see a new term being written with no corresponding vote for someone. It's possible that some node starts an election, writes term and vote for self, but due to applier being faster than relay on this node, it first delivers a term without vote through applier, and only then sends a term with vote through relay. Some remote candidate might schedule a vote for self instead of voting for the election starter in this case.
We've already fixed a similar issue for box.ctl.promote(): https://github.com/tarantool/tarantool/issues/8497. There term and vote were delivered separately even through relay.
Now the same issue may occur when delivering a term through applier (which was added in https://github.com/tarantool/tarantool/commit/544955109140e5dd5b956178fe47fd5ed3da2952).
Note that even though the normal relay channel delivers term and vote atomically (RAFT: message {term: 3, vote: 3, state: candidate, vclock: {1: 23}} from 3), it's already too late because a term was delivered alone via the applier channel (RAFT: received a newer term from 3)
[010] storage3 | 2024-05-22 10:24:55.731 [28221] main/126/main I> RAFT: bump term to 3, follow
[010] storage3 | 2024-05-22 10:24:55.731 [28221] main/126/main I> RAFT: vote for 3, follow
[010] storage3 | 2024-05-22 10:24:55.731 [28221] main/120/raft_worker I> RAFT: persisted state {term: 3, vote: 3}
[010] storage3 | 2024-05-22 10:24:55.731 [28221] main/120/raft_worker I> RAFT: enter candidate state with 1 self vote
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/125/main I> RAFT: received a newer term from 3
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/125/main I> RAFT: bump term to 3, follow
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/119/raft_worker I> RAFT: persisted state {term: 3}
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/119/raft_worker I> RAFT: vote for 2, follow
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/119/raft_worker I> RAFT: persisted state {term: 3, vote: 2}
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/119/raft_worker I> RAFT: enter candidate state with 1 self vote
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/114/applier/replicator@unix/:./storage3.iproto I> RAFT: message {term: 3, vote: 3, state: candidate, vclock: {1: 23}} from 3
[010] storage1 | 2024-05-22 10:24:55.731 [28189] main/114/applier/replicator@unix/:./storage3.iproto I> RAFT: vote request is skipped - competing candidate
What should be done
We need to think this through more thoroughly. There are various possible fixes. We can stop taking such term bumps in ACKs into account when deciding to start new elections, or maybe we can rethink raft communications between nodes, for example, make appliers send full raft state in ACKs or maybe completely detach raft communications from the data stream.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how the applier and relay deliver Raft term and vote state, then compare that behavior with the earlier box.ctl.promote fix in issue 8497 and commit 544955109140e5dd5b956178fe47fd5ed3da2952. The issue leaves several fixes open; done requires choosing and validating a design that prevents a term from being delivered without its corresponding vote.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- lua
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100