Cloud Auto-Join never rechecks metadata
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
`Nomad v0.10.3 (65af1b9ecff5b55a1dd6e10b8c3224f896d6c9fa)`
### Operating system and Environment details
```
$ uname -a
Linux ip-192-168-69-193 4.15.0-1058-aws #60-Ubuntu SMP Wed Jan 15 22:35:20 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
...
```
### Issue
We have nomad clients configured to use cloud autojoin with AWS
```
server_join {
retry_join = ["provider=aws tag_key=cluster_autojoin_key tag_value=consul-nomad-servers-bbuzbee-dev-gcs"]
}
```
This is a great feature, and on startup it works well. It enables us to use autoscaling groups to at will create servers and clients.
However it seems to me that the the client checks the cloud for server IPs only once at startup, and if they ever change t he agent has to be restarted. For example, in a cluster with 1 server (I know, 1 server is bad) I terminated the server and it was replaced with one with the correct tag, however the client sat for an hour and never discovered it
```
Mar 22 21:49:26 ip-192-168-69-193 nomad[3216]: {"@level":"error","@message":"error discovering nomad servers","@module":"client","@timestamp":"2020-03-22T21:49:26.776391Z","error":"client.consul: unable to query Consul datacenters: Unexpected response code: 500 (No known Consul servers)"}
Mar 22 21:49:49 ip-192-168-69-193 nomad[3216]: {"@level":"error","@message":"error performing RPC to server","@module":"client.rpc","@timestamp":"2020-03-22T21:49:49.737478Z","error":"rpc error: failed to get conn: dial tcp 192.168.61.156:4647: i/o timeout","rpc":"Node.GetClientAllocs","server":{"IP":"192.168.61.156","Port":4647,"Zone":""}}
Mar 22 21:49:49 ip-192-168-69-193 nomad[3216]: {"@level":"error","@message":"error querying node allocations","@module":"client","@timestamp":"2020-03-22T21:49:49.737555Z","error":"rpc error: failed to get conn: dial tcp 192.168.61.156:4647: i/o timeout"}
Mar 22 21:50:03 ip-192-168-69-193 nomad[3216]: {"@level":"error","@message":"error performing RPC to server","@module":"client.rpc","@timestamp":"2020-03-22T21:50:03.319344Z","error":"rpc error: failed to get conn: dial tcp 192.168.61.156:4647: i/o timeout","rpc":"Node.UpdateStatus","server":{"IP":"192.168.61.156","Port":4647,"Zone":""}}
Mar 22 21:50:03 ip-192-168-69-193 nomad[3216]: {"@level":"error","@message":"error heartbeating. retrying","@module":"client","@timestamp":"2020-03-22T21:50:03.319437Z","error":"failed to update status: rpc error: failed to get conn: dial tcp 192.168.61.156:4647: i/o timeout","period":27382646650}
```
All I had to do was restart nomad
`$ sudo systemctl restart nomad`
and it worked
```
Mar 22 21:50:32 ip-192-168-69-193 nomad[6925]: {"@level":"info","@message":"discover-aws: Instance i-0c6f880d5ffe7df87 has private ip 192.168.1.4","@module":"agent.joiner","@timestamp":"2020-03-22T21:50:32.631825Z"}
```
Would it not make sense to have nomad agents re-check the cloud instance metadata for servers if it is unable to connect for extended periods of time? This would save us a manual intervention step if all of the known server IPs change. Even in a cluster with more than 1 server I imagine all the known server IPs changing would leave us in this same state.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the server_join retry_join path through the agent.joiner and client components shown in the logs, focusing on when AWS discovery runs after startup. Reproduce a replaced server scenario and define done as discovering the new server IP without restarting Nomad, with coverage for the retry behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100