hashicorp / hashicorp/memberlist

memberlist fails in large-scale clusters

Open
#299 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4.1k
Forks
478
Avg merge
15h 58m
Merged PRs (30d)
3

Description

We recently implemented a large scale improved gossip protocol based on memberlist. memberlist did not meet our expectations when the number of nodes reached 1000. After investigation, it is found that the broadcast mechanism takes data from the system broadcast queue first(function `getBraodcasts` in broadcast.go). If the number of nodes is too large, the UDP packet does not have enough space to store the data in the user-defined broadcast queue, which leads to our failure to achieve system consistency.

Contributor guide

Open the contributing guide

Research direction

Start in broadcast.go at getBraodcasts and trace how system and user-defined broadcast queues are selected when UDP packet space is limited. Reproduce the behavior with a cluster near 1000 nodes and inspect how the broadcast path handles packet capacity. Done means the large-cluster broadcast mechanism no longer prevents system consistency.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.