Force restart of consul during startup results in zero byte node-id file
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
### `consul version` for both Client and Server
Client: `0.7.5`
Server: `0.7.5`
### `consul info` for both Client and Server
Client:
```
agent:
check_monitors = 2
check_ttls = 0
checks = 4
services = 3
build:
prerelease =
revision = '21f2d5a
version = 0.7.5
consul:
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 2
goroutines = 43
max_procs = 2
os = linux
version = go1.7.5
serf_lan:
encrypted = false
event_queue = 0
event_time = 8
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 13068
members = 62
query_queue = 0
query_time = 1
```
Server:
```
agent:
check_monitors = 1
check_ttls = 1
checks = 5
services = 6
build:
prerelease =
revision = '21f2d5a
version = 0.7.5
consul:
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 1.1.1.252:8300
server = true
raft:
applied_index = 9365
commit_index = 9365
fsm_pending = 0
last_contact = never
last_log_index = 9365
last_log_term = 2
last_snapshot_index = 8194
last_snapshot_term = 2
latest_configuration = [{Suffrage:Voter ID:1.1.1.252:8300 Address:1.1.1.252:8300} {Suffrage:Voter ID:1.1.1.253:8300 Address:1.1.1.253:8300} {Suffrage:Voter ID:1.1.1.254:8300 Address:1.1.1.254:8300}]
latest_configuration_index = 1
num_peers = 2
protocol_version = 1
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 4
goroutines = 286
max_procs = 4
os = linux
version = go1.7.5
serf_lan:
encrypted = false
event_queue = 0
event_time = 8
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 13068
members = 62
query_queue = 0
query_time = 1
serf_wan:
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```
### Operating system and Environment details
Server: Centos 7.2
Clients: Centos 7.2, Centos6.5, Ubuntu 14
### Description of the Issue (and unexpected/desired result)
From time to time consul fails to start with:
```
2017-09-21_17:56:34.26259 ==> Starting Consul agent...
2017-09-21_17:56:34.26373 ==> Error starting agent: Failed to setup node ID: uuid string is wrong length
```
Looking at the node-id file it is zero bytes. When we remove the node-id file and consul starts correctly.
Expected behavior: when consul starts the check to create the node-id file could be such that if the file doesn't exist or is empty, then create it.
### Reproduction steps
It appears this occurs when consul is in the process of starting and is force restarted mid-startup. This is a fairly rare failure likely due to how we deploy consul (generate the config files from templates and force-restart consul).
### Log Fragments
We run consul under runit so a majority of this output is from the run script. Looking at some other deploy logs I see runit force-restarting consul after which time consul fails to start.
```
2017-09-21_17:13:35.52594 + '[' -f /etc/sysconfig/consul ']'
2017-09-21_17:13:35.52599 + . /etc/sysconfig/consul
2017-09-21_17:13:35.52604 ++ INTERFACE=eth0
2017-09-21_17:13:35.52611 + '[' -f /etc/default/consul ']'
2017-09-21_17:13:35.52617 + CONSUL_FLAGS=
2017-09-21_17:13:35.52622 + INTERFACE=eth0
2017-09-21_17:13:35.52632 + for i in '$INTERFACE'
2017-09-21_17:13:35.52665 + ip -f inet addr show dev eth0 primary
2017-09-21_17:13:35.53182 + grep -Pq 'inet [0-9.]+'
2017-09-21_17:13:35.53280 ++ grep -Po 'inet \K[0-9.]+'
2017-09-21_17:13:35.53345 ++ ip -f inet addr show dev eth0 primary
2017-09-21_17:13:35.53421 ++ head -n 1
2017-09-21_17:13:35.53478 ++ grep -v /32
2017-09-21_17:13:35.53584 + BIND=1.1.1.26
2017-09-21_17:13:35.53585 + break
2017-09-21_17:13:35.53585 + '[' -z 1.1.1.26 ']'
2017-09-21_17:13:35.53999 ++ nproc
2017-09-21_17:13:35.73817 + export GOMAXPROCS=2
2017-09-21_17:13:35.73818 + GOMAXPROCS=2
2017-09-21_17:13:35.73819 + ulimit -n 65535
2017-09-21_17:13:35.73819 + exec nice -n -10 ionice -c 1 /usr/bin/consul agent -config-dir=/etc/tetration/consul.d/configs -bind=1.1.1.26 -advertise=1.1.1.26
2017-09-21_17:13:35.73819 ==> Starting Consul agent...
2017-09-21_17:13:45.74136 + '[' -f /etc/sysconfig/consul ']'
2017-09-21_17:13:45.74140 + . /etc/sysconfig/consul
2017-09-21_17:13:45.74144 ++ INTERFACE=eth0
2017-09-21_17:13:45.74148 + '[' -f /etc/default/consul ']'
2017-09-21_17:13:45.74152 + CONSUL_FLAGS=
2017-09-21_17:13:45.74155 + INTERFACE=eth0
2017-09-21_17:13:45.74162 + for i in '$INTERFACE'
2017-09-21_17:13:45.74182 + ip -f inet addr show dev eth0 primary
2017-09-21_17:13:45.74194 + grep -Pq 'inet [0-9.]+'
2017-09-21_17:13:45.74296 ++ ip -f inet addr show dev eth0 primary
2017-09-21_17:13:45.74335 ++ head -n 1
2017-09-21_17:13:45.74380 ++ grep -Po 'inet \K[0-9.]+'
2017-09-21_17:13:45.74392 ++ grep -v /32
2017-09-21_17:13:45.74489 + BIND=1.1.1.26
2017-09-21_17:13:45.74491 + break
2017-09-21_17:13:45.74492 + '[' -z 1.1.1.26 ']'
2017-09-21_17:13:45.74504 ++ nproc
2017-09-21_17:13:45.74557 + export GOMAXPROCS=2
2017-09-21_17:13:45.74558 + GOMAXPROCS=2
2017-09-21_17:13:45.74558 + ulimit -n 65535
2017-09-21_17:13:45.74559 + exec nice -n -10 ionice -c 1 /usr/bin/consul agent -config-dir=/etc/tetration/consul.d/configs -bind=1.1.1.26 -advertise=1.1.1.26
2017-09-21_17:13:45.76214 ==> Starting Consul agent...
2017-09-21_17:13:45.76303 ==> Error starting agent: Failed to setup node ID: uuid string is wrong length
```
Contributor guide
Assessment
This issue has not been assessed yet.