Auto-config node attempted to re-auto-config unsuccessfully
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
An agent node was set up with auto-config and successfully joined the cluster. It ran fine for more than a month.
At some point a networking issue meant that it became disconnected from the servers. When it went to reconnect it wasn't able to do so. It then appeared to run it's auto-config steps again but that failed because its auto-config token had expired.
It's not clear what in our logs is due to operator reboots and restarts and what is the agent behaviour. Please can you clarify what the auto-config behaviour is in cases of disconnection, and also if certificates or other target details have changed in that period? Happy to update the documentation pages accordingly (auto-config docs are quite light IMO)
One further specific question: we run the consul servers in an AWS auto-scaling group and use the discovery-by-tag features to look up the cluster. This means the IP addresses of the server nodes change. I noticed in the auto-config.json file in the data directory that the explicit ips of the server nodes are listed under RetryJoinLAN, rather than the auto-discovery details. Is it possible those are stale or are they not relevant at that point? Should both auto_config.server_addresses and retry_join config entries be specified in the main config file?
### Consul info for both Client and Server
Client and server are both 1.11.3
Servers are running on linux (ubuntu 20.04); client is running on windows server 2019
{
"ca_file": "",
"bind_addr": "0.0.0.0",
"disable_update_check": true,
"ports": {
"https": 8501,
"http": -1
},
"auto_config": {
"intro_token_file": "",
"server_addresses": [
"provider=aws tag_key= tag_value="
],
"enabled": true
},
"verify_server_hostname": false,
"verify_incoming": false,
"log_file": "",
"node_name": "",
"verify_incoming_rpc": true,
"server": false,
"client_addr": "127.0.0.1",
"verify_outgoing": true,
"advertise_addr": "{{ GetInterfaceIP \"eth0\" }}",
"datacenter": "",
"data_dir": "",
"ui_config": {
"enabled": false
}
}
Contributor guide
Research direction
Start with the auto-config.server_addresses setting and the generated auto-config.json entry for RetryJoinLAN, then review the reported reconnect and expired-token behavior in the Consul 1.11.3 configuration. Done means documenting what happens after disconnection, how changed certificates or server addresses are handled, and whether both configuration entries are needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, distributed-systems, documentation, networking
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100