FEATURE: Revisit retry-join value on failure
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
### `consul version` for both Client and Server
Client: `0.7.0`
Server: `0.7.0`
### Operating system and Environment details
AWS AMI
### Description of the Issue:
We have tested, and are using, the DNS entry of an ELB that is serving a cluster of Consul server instances in AWS created by an ASG for the `-retry-join` value of an agent. When the Consul agent container comes up, we pass the DNS name of the ELB to `retry-join` and the agent is properly seeded with the values of the Consul servers. It actually works great.
While testing for resiliency, we destroy the entire Consul cluster and see what happens to the agents. The agents return "No known Consul servers" while continuing to try to rejoin the set of IP addresses it originally received from the initial setup, but those instances no longer exist so the agent tries forever. If we simply restart the agent container, it uses `retry-join` again, finds the new cluster. It even reports the services it was managing, effectively healing itself...huzzah! As the agents are restarted one by one, the service catalog is re-populated and everyone is happy. (Our Consul servers self-populate their KV store upon initialization so that's taken care of by the LC of the ASG)
The feature request is as follows:
Under some set of circumstances, probably by configuration value, the agent will revisit the value in `-retry-join` instead of relying on its internal IP list while trying to phone home to the servers. This could happen after a timeout once "No known Consul servers" is found, or by setting something along the lines of "join_is_elb" or something better. Our setup would be 100% resilient to wipeouts or other catastrophes if Consul agent would simply re-examine the value given to it in `retry-join` at some point after exhausting its attempts at contacting all of the servers in its internal list.
Obviously this use case isn't for everyone, which is why it should be explicitly enabled and configurable.
Discuss.
Contributor guide
Research direction
Start by reproducing the AWS ELB and `-retry-join` scenario described in the issue, focusing on the agent behavior after it reports "No known Consul servers." Determine where retry-join values are initially resolved and define a configurable retry policy that re-examines the value after the internal server list is exhausted; done means agents can discover a replacement cluster without being restarted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100