redpanda-data / redpanda-data/redpanda

Integrate `node_status` information into Raft implementation

Open
#7,283 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/raft kind/enhance
Dominant language
C++
Stars
12.6k
Forks
792
PR merge metrics
No merged PRs in 30d

Description

Who is this for and what problem do they have today?

On systems with large partition counts heartbeats may be missed due to resource contention.
The user impact is that the corresponding partition is unavailable while the triggered election is in progress. It can also result in the connection transporting the heartbeat being terminated, which can have a knock on effect to the heartbeats of other Raft groups.

To address this problem a new node_status subsystem was introduced. All nodes periodically send node_status RPCs to all other nodes and node_status_table exposes the time point at which a node was last seen. This subsystem operates outside of Raft and it should not be impacted by slow-downs in the same way as Raft heartbeats.

What are the success criteria?

Node status information can be used in the Raft layer to make leadership more stable in two ways:

  1. When deciding whether to reconnect followers (consensus::should_reconnect_follower) we could check that the delta since the last node status is less than the raft timeout before counting the peer as a failure. We should be able to control how many times in a row we allow for this. There's a prototype for this in https://github.com/redpanda-data/redpanda/pull/6518, but it needs rebasing.
  2. When deciding whether to step down (consensus::maybe_step_down) because we've lost contact with the followers we could use the same approach as in (1). Again, we need to be able to control how many times this can happen in a row.
Why is solving this problem impactful?

Enables higher partition counts and makes raft leadership more stable.

Additional notes

Internal issue: here
PR implementing node status subsystem: https://github.com/redpanda-data/redpanda/pull/6440
Prototype for skipping re-connects: https://github.com/redpanda-data/redpanda/pull/6518
Node status RFC: https://docs.google.com/document/d/1OdAxmEHhVA3b6spC5JclGV9zOBlJ7-2jxT7RwIJ9ANI

JIRA Link: CORE-1082

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with consensus::should_reconnect_follower and consensus::maybe_step_down, then review the node_status subsystem and prototype in PR 6518. Compare the approach with the node status RFC and PR 6440; done means both Raft decisions use recent node-status information with configurable consecutive-failure limits and leadership is more stable.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.