hashicorp / hashicorp/consul

Auto-config node attempted to re-auto-config unsuccessfully

Open
#12,849 2 comments 0 reactions 0 assignees View on GitHub
theme/config type/docs type/question
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

An agent node was set up with auto-config and successfully joined the cluster. It ran fine for more than a month.

At some point a networking issue meant that it became disconnected from the servers. When it went to reconnect it wasn't able to do so. It then appeared to run it's auto-config steps again but that failed because its auto-config token had expired.

It's not clear what in our logs is due to operator reboots and restarts and what is the agent behaviour. Please can you clarify what the auto-config behaviour is in cases of disconnection, and also if certificates or other target details have changed in that period? Happy to update the documentation pages accordingly (auto-config docs are quite light IMO)

One further specific question: we run the consul servers in an AWS auto-scaling group and use the discovery-by-tag features to look up the cluster. This means the IP addresses of the server nodes change. I noticed in the auto-config.json file in the data directory that the explicit ips of the server nodes are listed under RetryJoinLAN, rather than the auto-discovery details. Is it possible those are stale or are they not relevant at that point? Should both auto_config.server_addresses and retry_join config entries be specified in the main config file?

### Consul info for both Client and Server

Client and server are both 1.11.3
Servers are running on linux (ubuntu 20.04); client is running on windows server 2019

{
"ca_file": "",
"bind_addr": "0.0.0.0",
"disable_update_check": true,
"ports": {
"https": 8501,
"http": -1
},
"auto_config": {
"intro_token_file": "",
"server_addresses": [
"provider=aws tag_key= tag_value="
],
"enabled": true
},
"verify_server_hostname": false,
"verify_incoming": false,
"log_file": "",
"node_name": "",
"verify_incoming_rpc": true,
"server": false,
"client_addr": "127.0.0.1",
"verify_outgoing": true,
"advertise_addr": "{{ GetInterfaceIP \"eth0\" }}",
"datacenter": "",
"data_dir": "",
"ui_config": {
"enabled": false
}
}

Contributor guide

Open the contributing guide

Research direction

Start with the auto-config.server_addresses setting and the generated auto-config.json entry for RetryJoinLAN, then review the reported reconnect and expired-token behavior in the Consul 1.11.3 configuration. Done means documenting what happens after disconnection, how changed certificates or server addresses are handled, and whether both configuration entries are needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, distributed-systems, documentation, networking
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.