hashicorp / hashicorp/envconsul

Prevent concurrent restart

Open
#173 0 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
2.1k
Forks
194
PR merge metrics
No merged PRs in 30d

Description

Currently there is no way to ensure only N of M servers running the same process restart after a config change concurrently.

There is a splay option to add a random delay which can partially address this issue, but randomness is a part of it and it is ill suited to processes that are slow to start.

As mentioned in #53 using a Consul lock could be used to implement this reliably with the lock needing to be acquired before the process is restarted (the lock would not be used during startup, only during a restart/reload after config change).

The lock should be combined with a fixed timeout (at a minimum as a failover, but also in case there isn't any good way to signal that the service is "ready") combined optionally with other checks such as that the process has started listening on a certain port, or looking for a certain stdout/stderr line. Perhaps Consul's own service health checking could be utilised in some fashion to signal this. This would be a once off check to release the lock (unless the timeout happens first).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.