moby / moby/swarmkit

Race in manager startup

Open
#2,462 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/raft kind/bug
Dominant language
Go
Stars
3.7k
Forks
676
Avg merge
4d 9h
Merged PRs (30d)
6

Description

Manager.Run() calls handleLeadershipEvents() as a go routine and may access objects in the store before the store is completely initialized. The race is with raft.Node.Run(), which is responsible for applying the raft log to the store.

An example is KeyManager init which happens within handleLeadershipEvents(), which does a lookup of the cluster object. If the raft log is big and modifies the cluster object, the keymanager init can possibly fail.

There might be other such cases. Consider waiting for the raftlog to be applied first before performing other startup actions.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing Manager.Run(), handleLeadershipEvents(), and raft.Node.Run() to understand their startup ordering and how the raft log populates the store. Reproduce or inspect the KeyManager initialization path and its cluster lookup. Done means startup actions cannot read partially initialized store state, and the race scenario is covered by appropriate verification.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.