hashicorp / hashicorp/serf

Better handling of snapshot replays and events

Open
#465 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
6.1k
Forks
609
Avg merge
15h 1m
Merged PRs (30d)
3

Description

When you do an initial join, Serf will do a push/pull of the state from the other nodes, which does include the buffer of events that node has experienced. The https://www.serf.io/docs/agent/options.html#replay governs whether those play out or not.

What's interesting is that Serf can do a join at startup with some other node before it is attempting to perform the cluster join on its own:

2017/04/19 11:47:48 [INFO] agent: joining: [] replay: false

That "replay:false" indicates that the replay configuration is set correctly. I don't think the replay setting applies to any other push/pull, and `` appears earlier in the snapshot before the events, so the sequence of events looks like this:

1. Agent starts up and begins running through its snapshot
2. Adding a peer of `` works, and Serf decides to push/pull
3. The push/pull with `` lets the restarted agent know about the event buffer
4. The events replay since the snapshot replay hasn't yet encountered the event clock statements that would make it ignore the events
5. The agent joins `` well after all this happens, so the replay setting doesn't guard anything.

It seems like we should probably scan the snapshot and set the event clock basically before we do anything else (even accept incoming i/o from peers). That would prevent any previous events from firing. I think if the node was part of an existing cluster you'd run into trouble even if you delete the snapshot to try to prevent this, because an existing node in the cluster could choose to do an incoming push/pull which would dump the events as well, making it skip the "replay:false".

We could also make the replay setting more universal and apply it to push/pull operations as well, since you almost never want to ingest events way after the fact.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.