moby / moby/swarmkit

Applying raft logs blocks heartbeating

Open
#1,346 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/raft priority/P3
Dominant language
Go
Stars
3.7k
Forks
676
Avg merge
4d 9h
Merged PRs (30d)
6

Description

A raft node can't send heartbeats while it's applying logs or snapshots. If a log or snapshot is large enough, the node could be treated as down, possibly causing loss of quorum.

We should fix this by applying logs asynchronously and calling Advance when finished.

This is probably too late for 1.12.1. Not sure we want to change it in a patch release anyway.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the Raft log and snapshot application path to identify where applying blocks heartbeat handling and where Advance is currently called. Read the surrounding Raft node lifecycle before assessing asynchronous application. Done means heartbeats continue during large applies and Advance occurs after application finishes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.