Nomad binds to unexpected interface
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
Hi there,
after upgrading nomad to 1.0.4, starting nomad via systemd fails to elect leader.
Starting via `nomad agent -config /etc/nomad.d` continues to work fine. Stopping and starting after a reboot via `systemctl stop nomad && systemctl start nomad` works as well.
Why is the leader election just failing after system restart?
I am running nomad in single server mode. Nomad was installed via apt.
### Nomad version
`Nomad v1.0.4 (9294f35f9aa8dbb4acb6e85fa88e3e2534a3e41a)`
### nomad.hcl
```
root@server-0:~# cat /etc/nomad.d/nomad.hcl
datacenter = "mode"
data_dir = "/opt/nomad/data"
bind_addr = "0.0.0.0"
server {
enabled = true
bootstrap_expect = 1
}
client {
enabled = true
servers = ["127.0.0.1:4646"]
network_interface = "ens10"
template {
disable_file_sandbox = true
}
host_volume "rabbitmq" {
path = "/opt/rabbitmq"
read_only = false
}
host_volume "arangodb" {
path = "/opt/arangodb"
read_only = false
}
options {
"docker.auth.config" = "/root/.docker/config.json"
}
}
acl {
enabled = true
}
```
### nomad.service
```sh
root@server-0:/etc/systemd/system/multi-user.target.wants# cat nomad.service
[Unit]
Description=Nomad
Documentation=https://nomadproject.io/docs/
Wants=network-online.target
After=network-online.target
# When using Nomad with Consul it is not necessary to start Consul first. These
# lines start Consul before Nomad as an optimization to avoid Nomad logging
# that Consul is unavailable at startup.
#Wants=consul.service
#After=consul.service
[Service]
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/bin/nomad agent -config /etc/nomad.d
KillMode=process
KillSignal=SIGINT
LimitNOFILE=65536
LimitNPROC=infinity
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitInterval=10
TasksMax=infinity
OOMScoreAdjust=-1000
[Install]
WantedBy=multi-user.target
```
### Operating system and Environment details
Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-66-generic x86_64)
### Logs after reboot
```sh
root@server-0:/etc/systemd/system/multi-user.target.wants# journalctl -xef -u nomad
Mar 13 18:24:26 server-0 nomad[526]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
Mar 13 18:24:26 server-0 nomad[526]: ==> Loaded configuration from /etc/nomad.d/nomad.hcl
Mar 13 18:24:26 server-0 nomad[526]: ==> Starting Nomad agent...
Mar 13 18:24:28 server-0 nomad[526]: ==> Nomad agent configuration:
Mar 13 18:24:28 server-0 nomad[526]: Advertise Addrs: HTTP: 10.0.1.1:4646; RPC: 10.0.1.1:4647; Serf: 10.0.1.1:4648
Mar 13 18:24:28 server-0 nomad[526]: Bind Addrs: HTTP: 0.0.0.0:4646; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
Mar 13 18:24:28 server-0 nomad[526]: Client: true
Mar 13 18:24:28 server-0 nomad[526]: Log Level: INFO
Mar 13 18:24:28 server-0 nomad[526]: Region: global (DC: mode)
Mar 13 18:24:28 server-0 nomad[526]: Server: true
Mar 13 18:24:28 server-0 nomad[526]: Version: 1.0.4
Mar 13 18:24:28 server-0 nomad[526]: ==> Nomad agent started! Log data will stream in below:
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.356+0100 [WARN] agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/opt/nomad/data/plugins
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.364+0100 [INFO] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.364+0100 [INFO] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.364+0100 [INFO] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.364+0100 [INFO] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.364+0100 [INFO] agent: detected plugin: name=java type=driver plugin_version=0.1.0
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.364+0100 [INFO] agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.399+0100 [INFO] nomad.raft: restored from snapshot: id=5-139292-1615545712266
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.521+0100 [INFO] nomad.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:172.17.0.1:4647 Address:172.17.0.1:4647}]"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.521+0100 [INFO] nomad.raft: entering follower state: follower="Node at 10.0.1.1:4647 [Follower]" leader=
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.523+0100 [INFO] nomad: serf: EventMemberJoin: server-0.global 10.0.1.1
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.523+0100 [INFO] nomad: starting scheduling worker(s): num_workers=2 schedulers=[service, batch, system, _core]
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.523+0100 [WARN] nomad: serf: Failed to re-join any previously known node
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.524+0100 [INFO] client: using state directory: state_dir=/opt/nomad/data/client
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.524+0100 [INFO] nomad: adding server: server="server-0.global (Addr: 10.0.1.1:4647) (DC: mode)"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.525+0100 [INFO] client: using alloc directory: alloc_dir=/opt/nomad/data/alloc
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.537+0100 [INFO] client.fingerprint_mgr.cgroup: cgroups are available
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.547+0100 [WARN] client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=ens10
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.548+0100 [WARN] client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=lo
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.550+0100 [WARN] client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=eth0
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.554+0100 [WARN] client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=ens10
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.570+0100 [INFO] client.plugin: starting plugin manager: plugin-type=csi
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.571+0100 [INFO] client.plugin: starting plugin manager: plugin-type=driver
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:26.572+0100 [INFO] client.plugin: starting plugin manager: plugin-type=device
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:27.624+0100 [WARN] nomad.raft: not part of stable configuration, aborting election
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.367+0100 [ERROR] client.rpc: error performing RPC to server: error="rpc error: failed to get conn: dial tcp 127.0.0.1:4646: connect: connection refused" rpc=Node.Register server=127.0.0.1:4646
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.367+0100 [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: failed to get conn: dial tcp 127.0.0.1:4646: connect: connection refused" rpc=Node.Register server=127.0.0.1:4646
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.367+0100 [ERROR] client: error registering: error="rpc error: failed to get conn: dial tcp 127.0.0.1:4646: connect: connection refused"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.403+0100 [WARN] client.driver_mgr.docker: failed to reattach to docker logger process: driver=docker error="failed to reattach to docker logger process: Reattachment process not found"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.423+0100 [ERROR] client.driver_mgr.docker.docker_logger.nomad: log streaming ended with terminal error: driver=docker @module=docker_logger error="open /opt/nomad/data/alloc/2b0a535b-31b5-536a-5f1f-fce28ab9039d/alloc/logs/.arangodb.stdout.fifo: no such file or directory" timestamp=2021-03-13T18:24:28.423+0100
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.449+0100 [INFO] client: started client: node_id=20843003-7c06-2b05-8e29-090a1a5e118f
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.454+0100 [INFO] client.gc: marking allocation for GC: alloc_id=1dd29296-2043-7845-7fd9-ad9c6401338d
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.455+0100 [INFO] client.gc: marking allocation for GC: alloc_id=3e4cd59e-8021-deea-7bef-145c08b916d0
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.455+0100 [INFO] client.gc: marking allocation for GC: alloc_id=9ef77798-f8de-f31f-d786-89378fb82b45
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.455+0100 [INFO] client.gc: marking allocation for GC: alloc_id=e6c2f195-fed5-0758-3d5c-4d4dc4e31c54
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.455+0100 [INFO] client.gc: marking allocation for GC: alloc_id=fb6c61e9-bea7-5371-ddde-9737856c6939
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.455+0100 [INFO] client.gc: marking allocation for GC: alloc_id=0e0513cc-1ea3-2b0a-dbaf-14730684a0d4
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.455+0100 [INFO] client.gc: marking allocation for GC: alloc_id=14cd9970-f200-8893-d712-11d6741db11b
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.455+0100 [INFO] client.gc: marking allocation for GC: alloc_id=167462ef-95cd-82fd-3c18-ce18be9f2c9d
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.455+0100 [INFO] client.gc: marking allocation for GC: alloc_id=2264563b-b2a6-86fc-0d2c-8f6c193c05f3
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.456+0100 [INFO] client.gc: marking allocation for GC: alloc_id=4b64d23a-2fcf-0b28-75a5-26c8d4d01ef4
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.456+0100 [INFO] client.gc: marking allocation for GC: alloc_id=62cdb29f-9bfc-0aef-c629-eca29da4fda7
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.456+0100 [INFO] client.gc: marking allocation for GC: alloc_id=7e010a27-a485-5ac9-8b61-73cd3997bbf2
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.458+0100 [WARN] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=2b0a535b-31b5-536a-5f1f-fce28ab9039d task=arangodb error="Reattachment process not found"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.472+0100 [INFO] client.gc: marking allocation for GC: alloc_id=2b0a535b-31b5-536a-5f1f-fce28ab9039d
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.472+0100 [INFO] client.gc: garbage collecting allocation: alloc_id=3e4cd59e-8021-deea-7bef-145c08b916d0 reason="forced collection"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.473+0100 [INFO] client.gc: garbage collecting allocation: alloc_id=62cdb29f-9bfc-0aef-c629-eca29da4fda7 reason="forced collection"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.473+0100 [INFO] client.gc: garbage collecting allocation: alloc_id=2b0a535b-31b5-536a-5f1f-fce28ab9039d reason="forced collection"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.473+0100 [INFO] client.gc: garbage collecting allocation: alloc_id=9ef77798-f8de-f31f-d786-89378fb82b45 reason="forced collection"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.473+0100 [INFO] client.gc: garbage collecting allocation: alloc_id=fb6c61e9-bea7-5371-ddde-9737856c6939 reason="forced collection"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.473+0100 [INFO] client.gc: garbage collecting allocation: alloc_id=4b64d23a-2fcf-0b28-75a5-26c8d4d01ef4 reason="forced collection"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.473+0100 [INFO] client.gc: garbage collecting allocation: alloc_id=2264563b-b2a6-86fc-0d2c-8f6c193c05f3 reason="forced collection"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.473+0100 [INFO] client.gc: garbage collecting allocation: alloc_id=1dd29296-2043-7845-7fd9-ad9c6401338d reason="forced collection"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.473+0100 [INFO] client.gc: garbage collecting allocation: alloc_id=7e010a27-a485-5ac9-8b61-73cd3997bbf2 reason="forced collection"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.473+0100 [INFO] client.gc: garbage collecting allocation: alloc_id=0e0513cc-1ea3-2b0a-dbaf-14730684a0d4 reason="forced collection"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.485+0100 [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=2b0a535b-31b5-536a-5f1f-fce28ab9039d task=arangodb @module=logmon path=/opt/nomad/data/alloc/2b0a535b-31b5-536a-5f1f-fce28ab9039d/alloc/logs/.arangodb.stdout.fifo timestamp=2021-03-13T18:24:28.484+0100
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.485+0100 [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=2b0a535b-31b5-536a-5f1f-fce28ab9039d task=arangodb @module=logmon path=/opt/nomad/data/alloc/2b0a535b-31b5-536a-5f1f-fce28ab9039d/alloc/logs/.arangodb.stderr.fifo timestamp=2021-03-13T18:24:28.485+0100
Mar 13 18:24:28 server-0 nomad[526]: 2021/03/13 18:24:28.498590 [INFO] (runner) creating new runner (dry: false, once: false)
Mar 13 18:24:28 server-0 nomad[526]: 2021/03/13 18:24:28.499534 [INFO] (runner) creating watcher
Mar 13 18:24:28 server-0 nomad[526]: 2021/03/13 18:24:28.501957 [INFO] (runner) starting
Mar 13 18:24:28 server-0 nomad[526]: 2021/03/13 18:24:28.505304 [INFO] (runner) rendered "/root/env/arangodb.env" => "/opt/nomad/data/alloc/2b0a535b-31b5-536a-5f1f-fce28ab9039d/arangodb/arangodb.env"
Mar 13 18:24:28 server-0 nomad[526]: 2021-03-13T18:24:28.534+0100 [WARN] client.driver_mgr.docker: RemoveImage on non-referenced counted image id: driver=docker image_id=sha256:972479e15e2d87f60f97b430bdb424f3a85a0504965b1f1d6ef1b0ff09581920
Mar 13 18:24:32 server-0 nomad[526]: 2021-03-13T18:24:32.548+0100 [WARN] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=2b0a535b-31b5-536a-5f1f-fce28ab9039d task=arangodb @module=logmon timestamp=2021-03-13T18:24:32.548+0100
Mar 13 18:24:32 server-0 nomad[526]: 2021-03-13T18:24:32.549+0100 [WARN] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=2b0a535b-31b5-536a-5f1f-fce28ab9039d task=arangodb @module=logmon timestamp=2021-03-13T18:24:32.548+0100
Mar 13 18:24:32 server-0 nomad[526]: 2021/03/13 18:24:32.554773 [INFO] (runner) stopping
Mar 13 18:24:32 server-0 nomad[526]: 2021/03/13 18:24:32.554955 [INFO] (runner) received finish
Mar 13 18:24:33 server-0 nomad[526]: 2021-03-13T18:24:33.495+0100 [ERROR] client.rpc: error performing RPC to server: error="rpc error: No cluster leader" rpc=Node.Register server=0.0.0.0:4647
Mar 13 18:24:33 server-0 nomad[526]: 2021-03-13T18:24:33.496+0100 [ERROR] client.rpc: error performing RPC to server, deadline exceeded, cannot retry: error="rpc error: No cluster leader" rpc=Node.Register server=0.0.0.0:4647
Mar 13 18:24:33 server-0 nomad[526]: 2021-03-13T18:24:33.496+0100 [ERROR] client: error registering: error="rpc error: No cluster leader"
Mar 13 18:24:33 server-0 nomad[526]: 2021-03-13T18:24:33.704+0100 [ERROR] client.rpc: error performing RPC to server: error="rpc error: No cluster leader" rpc=Node.UpdateAlloc server=0.0.0.0:4647
Mar 13 18:24:33 server-0 nomad[526]: 2021-03-13T18:24:33.704+0100 [ERROR] client.rpc: error performing RPC to server, deadline exceeded, cannot retry: error="rpc error: No cluster leader" rpc=Node.UpdateAlloc server=0.0.0.0:4647
Mar 13 18:24:33 server-0 nomad[526]: 2021-03-13T18:24:33.704+0100 [ERROR] client: error updating allocations: error="rpc error: No cluster leader"
Mar 13 18:24:36 server-0 nomad[526]: 2021-03-13T18:24:36.762+0100 [ERROR] worker: failed to dequeue evaluation: error="No cluster leader"
Mar 13 18:24:36 server-0 nomad[526]: 2021-03-13T18:24:36.808+0100 [ERROR] worker: failed to dequeue evaluation: error="No cluster leader"
Mar 13 18:24:41 server-0 nomad[526]: 2021-03-13T18:24:41.943+0100 [ERROR] worker: failed to dequeue evaluation: error="No cluster leader"
Mar 13 18:24:41 server-0 nomad[526]: 2021-03-13T18:24:41.950+0100 [ERROR] worker: failed to dequeue evaluation: error="No cluster leader"
Mar 13 18:24:46 server-0 nomad[526]: 2021-03-13T18:24:46.590+0100 [ERROR] client.rpc: error performing RPC to server: error="rpc error: No cluster leader" rpc=Node.UpdateAlloc server=10.0.1.1:4647
Mar 13 18:24:46 server-0 nomad[526]: 2021-03-13T18:24:46.592+0100 [ERROR] client.rpc: error performing RPC to server, deadline exceeded, cannot retry: error="rpc error: No cluster leader" rpc=Node.UpdateAlloc server=10.0.1.1:4647
Mar 13 18:24:46 server-0 nomad[526]: 2021-03-13T18:24:46.592+0100 [ERROR] client: error updating allocations: error="rpc error: No cluster leader"
Mar 13 18:24:47 server-0 nomad[526]: 2021-03-13T18:24:47.279+0100 [ERROR] worker: failed to dequeue evaluation: error="No cluster leader"
Mar 13 18:24:47 server-0 nomad[526]: 2021-03-13T18:24:47.341+0100 [ERROR] worker: failed to dequeue evaluation: error="No cluster leader"
Mar 13 18:24:53 server-0 nomad[526]: 2021-03-13T18:24:53.545+0100 [ERROR] client: yamux: Invalid protocol version: 72
Mar 13 18:24:53 server-0 nomad[526]: 2021-03-13T18:24:53.546+0100 [ERROR] client: yamux: Failed to write header: write tcp 127.0.0.1:59638->127.0.0.1:4646: write: connection reset by peer
Mar 13 18:24:53 server-0 nomad[526]: 2021-03-13T18:24:53.546+0100 [ERROR] client.rpc: error performing RPC to server: error="rpc error: msgpack encode error: write tcp 127.0.0.1:59638->127.0.0.1:4646: write: connection reset by peer" rpc=Node.UpdateAlloc server=127.0.0.1:4646
Mar 13 18:24:53 server-0 nomad[526]: 2021-03-13T18:24:53.546+0100 [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: msgpack encode error: write tcp 127.0.0.1:59638->127.0.0.1:4646: write: connection reset by peer" rpc=Node.UpdateAlloc server=127.0.0.1:4646
Mar 13 18:24:53 server-0 nomad[526]: 2021-03-13T18:24:53.546+0100 [ERROR] client: error updating allocations: error="rpc error: msgpack encode error: write tcp 127.0.0.1:59638->127.0.0.1:4646: write: connection reset by peer"
Mar 13 18:24:53 server-0 nomad[526]: 2021-03-13T18:24:53.628+0100 [ERROR] worker: failed to dequeue evaluation: error="No cluster leader"
Mar 13 18:24:53 server-0 nomad[526]: 2021-03-13T18:24:53.788+0100 [ERROR] worker: failed to dequeue evaluation: error="No cluster leader"
Mar 13 18:24:59 server-0 nomad[526]: 2021-03-13T18:24:59.729+0100 [ERROR] client.rpc: error performing RPC to server: error="rpc error: No cluster leader" rpc=Node.Register server=0.0.0.0:4647
Mar 13 18:24:59 server-0 nomad[526]: 2021-03-13T18:24:59.729+0100 [ERROR] client.rpc: error performing RPC to server, deadline exceeded, cannot retry: error="rpc error: No cluster leader" rpc=Node.Register server=0.0.0.0:4647
Mar 13 18:24:59 server-0 nomad[526]: 2021-03-13T18:24:59.729+0100 [ERROR] client: error registering: error="rpc error: No cluster leader"
Mar 13 18:25:03 server-0 nomad[526]: 2021-03-13T18:25:03.828+0100 [ERROR] worker: failed to dequeue evaluation: error="No cluster leader"
Mar 13 18:25:04 server-0 nomad[526]: 2021-03-13T18:25:04.161+0100 [ERROR] worker: failed to dequeue evaluation: error="No cluster leader"
Mar 13 18:25:07 server-0 nomad[526]: 2021-03-13T18:25:07.060+0100 [ERROR] client.rpc: error performing RPC to server: error="rpc error: No cluster leader" rpc=Node.UpdateAlloc server=10.0.1.1:4647
Mar 13 18:25:07 server-0 nomad[526]: 2021-03-13T18:25:07.060+0100 [ERROR] client.rpc: error performing RPC to server, deadline exceeded, cannot retry: error="rpc error: No cluster leader" rpc=Node.UpdateAlloc server=10.0.1.1:4647
Mar 13 18:25:07 server-0 nomad[526]: 2021-03-13T18:25:07.060+0100 [ERROR] client: error updating allocations: error="rpc error: No cluster leader"
Mar 13 18:25:14 server-0 nomad[526]: 2021-03-13T18:25:14.510+0100 [ERROR] client: yamux: Invalid protocol version: 72
Mar 13 18:25:14 server-0 nomad[526]: 2021-03-13T18:25:14.510+0100 [ERROR] client: yamux: Failed to write body: write tcp 127.0.0.1:59646->127.0.0.1:4646: write: connection reset by peer
Mar 13 18:25:14 server-0 nomad[526]: 2021-03-13T18:25:14.510+0100 [ERROR] client.rpc: error performing RPC to server: error="rpc error: msgpack encode error: session shutdown" rpc=Node.UpdateAlloc server=127.0.0.1:4646
Mar 13 18:25:14 server-0 nomad[526]: 2021-03-13T18:25:14.510+0100 [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: msgpack encode error: session shutdown" rpc=Node.UpdateAlloc server=127.0.0.1:4646
Mar 13 18:25:14 server-0 nomad[526]: 2021-03-13T18:25:14.510+0100 [ERROR] client: error updating allocations: error="rpc error: msgpack encode error: session shutdown"
Mar 13 18:25:18 server-0 nomad[526]: 2021-03-13T18:25:18.508+0100 [ERROR] worker: failed to dequeue evaluation: error="No cluster leader"
Mar 13 18:25:18 server-0 nomad[526]: 2021-03-13T18:25:18.632+0100 [ERROR] worker: failed to dequeue evaluation: error="No cluster leader"
Mar 13 18:25:19 server-0 nomad[526]: 2021-03-13T18:25:19.556+0100 [ERROR] client.rpc: error performing RPC to server: error="rpc error: No cluster leader" rpc=Node.Register server=0.0.0.0:4647
Mar 13 18:25:19 server-0 nomad[526]: 2021-03-13T18:25:19.556+0100 [ERROR] client.rpc: error performing RPC to server, deadline exceeded, cannot retry: error="rpc error: No cluster leader" rpc=Node.Register server=0.0.0.0:4647
Mar 13 18:25:19 server-0 nomad[526]: 2021-03-13T18:25:19.556+0100 [ERROR] client: error registering: error="rpc error: No cluster leader"
Mar 13 18:25:27 server-0 nomad[526]: 2021-03-13T18:25:27.129+0100 [ERROR] client.rpc: error performing RPC to server: error="rpc error: No cluster leader" rpc=Node.UpdateAlloc server=10.0.1.1:4647
Mar 13 18:25:27 server-0 nomad[526]: 2021-03-13T18:25:27.129+0100 [ERROR] client.rpc: error performing RPC to server, deadline exceeded, cannot retry: error="rpc error: No cluster leader" rpc=Node.UpdateAlloc server=10.0.1.1:4647
Mar 13 18:25:27 server-0 nomad[526]: 2021-03-13T18:25:27.129+0100 [ERROR] client: error updating allocations: error="rpc error: No cluster leader"
```
### Logs after restarting via `systemctl stop nomad && systemctl start nomad`
```sh
Mar 13 18:28:55 server-0 nomad[1092]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
Mar 13 18:28:55 server-0 nomad[1092]: ==> Loaded configuration from /etc/nomad.d/nomad.hcl
Mar 13 18:28:55 server-0 nomad[1092]: ==> Starting Nomad agent...
Mar 13 18:28:55 server-0 nomad[1092]: ==> Nomad agent configuration:
Mar 13 18:28:55 server-0 nomad[1092]: Advertise Addrs: HTTP: 172.17.0.1:4646; RPC: 172.17.0.1:4647; Serf: 172.17.0.1:4648
Mar 13 18:28:55 server-0 nomad[1092]: Bind Addrs: HTTP: 0.0.0.0:4646; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
Mar 13 18:28:55 server-0 nomad[1092]: Client: true
Mar 13 18:28:55 server-0 nomad[1092]: Log Level: INFO
Mar 13 18:28:55 server-0 nomad[1092]: Region: global (DC: mode)
Mar 13 18:28:55 server-0 nomad[1092]: Server: true
Mar 13 18:28:55 server-0 nomad[1092]: Version: 1.0.4
Mar 13 18:28:55 server-0 nomad[1092]: ==> Nomad agent started! Log data will stream in below:
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.194+0100 [WARN] agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/opt/nomad/data/plugins
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.198+0100 [INFO] agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.199+0100 [INFO] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.199+0100 [INFO] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.199+0100 [INFO] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.199+0100 [INFO] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.199+0100 [INFO] agent: detected plugin: name=java type=driver plugin_version=0.1.0
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.208+0100 [INFO] nomad.raft: restored from snapshot: id=5-139292-1615545712266
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.226+0100 [INFO] nomad.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:172.17.0.1:4647 Address:172.17.0.1:4647}]"
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.226+0100 [INFO] nomad.raft: entering follower state: follower="Node at 172.17.0.1:4647 [Follower]" leader=
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.228+0100 [INFO] nomad: serf: EventMemberJoin: server-0.global 172.17.0.1
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.229+0100 [INFO] nomad: starting scheduling worker(s): num_workers=2 schedulers=[service, batch, system, _core]
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.229+0100 [INFO] client: using state directory: state_dir=/opt/nomad/data/client
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.229+0100 [INFO] client: using alloc directory: alloc_dir=/opt/nomad/data/alloc
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.229+0100 [INFO] nomad: adding server: server="server-0.global (Addr: 172.17.0.1:4647) (DC: mode)"
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.229+0100 [WARN] nomad: serf: Failed to re-join any previously known node
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.231+0100 [INFO] client.fingerprint_mgr.cgroup: cgroups are available
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.235+0100 [WARN] client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=ens10
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.237+0100 [WARN] client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=lo
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.238+0100 [WARN] client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=eth0
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.239+0100 [WARN] client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=ens10
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.240+0100 [WARN] client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=docker0
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.247+0100 [INFO] client.plugin: starting plugin manager: plugin-type=csi
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.248+0100 [INFO] client.plugin: starting plugin manager: plugin-type=driver
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.248+0100 [INFO] client.plugin: starting plugin manager: plugin-type=device
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.302+0100 [ERROR] client.rpc: error performing RPC to server: error="rpc error: failed to get conn: dial tcp 127.0.0.1:4646: connect: connection refused" rpc=Node.Register server=127.0.0.1:4646
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.302+0100 [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: failed to get conn: dial tcp 127.0.0.1:4646: connect: connection refused" rpc=Node.Register server=127.0.0.1:4646
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.302+0100 [ERROR] client: error registering: error="rpc error: failed to get conn: dial tcp 127.0.0.1:4646: connect: connection refused"
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.306+0100 [INFO] client: started client: node_id=20843003-7c06-2b05-8e29-090a1a5e118f
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.310+0100 [INFO] client.gc: marking allocation for GC: alloc_id=e6c2f195-fed5-0758-3d5c-4d4dc4e31c54
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.311+0100 [INFO] client.gc: marking allocation for GC: alloc_id=14cd9970-f200-8893-d712-11d6741db11b
Mar 13 18:28:55 server-0 nomad[1092]: 2021-03-13T18:28:55.312+0100 [INFO] client.gc: marking allocation for GC: alloc_id=167462ef-95cd-82fd-3c18-ce18be9f2c9d
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.222+0100 [WARN] nomad.raft: heartbeat timeout reached, starting election: last-leader=
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.222+0100 [INFO] nomad.raft: entering candidate state: node="Node at 172.17.0.1:4647 [Candidate]" term=11
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.225+0100 [INFO] nomad.raft: election won: tally=1
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.225+0100 [INFO] nomad.raft: entering leader state: leader="Node at 172.17.0.1:4647 [Leader]"
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.226+0100 [INFO] nomad: cluster leadership acquired
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.325+0100 [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=4b64d23a-2fcf-0b28-75a5-26c8d4d01ef4 task=arangodb @module=logmon path=/opt/nomad/data/alloc/4b64d23a-2fcf-0b28-75a5-26c8d4d01ef4/alloc/logs/.arangodb.stdout.fifo timestamp=2021-03-13T18:28:57.323+0100
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.325+0100 [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=4b64d23a-2fcf-0b28-75a5-26c8d4d01ef4 task=arangodb @module=logmon path=/opt/nomad/data/alloc/4b64d23a-2fcf-0b28-75a5-26c8d4d01ef4/alloc/logs/.arangodb.stderr.fifo timestamp=2021-03-13T18:28:57.323+0100
Mar 13 18:28:57 server-0 nomad[1092]: 2021/03/13 18:28:57.363967 [INFO] (runner) creating new runner (dry: false, once: false)
Mar 13 18:28:57 server-0 nomad[1092]: 2021/03/13 18:28:57.364819 [INFO] (runner) creating watcher
Mar 13 18:28:57 server-0 nomad[1092]: 2021/03/13 18:28:57.365174 [INFO] (runner) starting
Mar 13 18:28:57 server-0 nomad[1092]: 2021/03/13 18:28:57.367497 [INFO] (runner) rendered "/root/env/arangodb.env" => "/opt/nomad/data/alloc/4b64d23a-2fcf-0b28-75a5-26c8d4d01ef4/arangodb/arangodb.env"
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.367+0100 [INFO] client.gc: marking allocation for GC: alloc_id=3e4cd59e-8021-deea-7bef-145c08b916d0
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.368+0100 [INFO] client.gc: marking allocation for GC: alloc_id=fb6c61e9-bea7-5371-ddde-9737856c6939
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.391+0100 [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=9ef77798-f8de-f31f-d786-89378fb82b45 task=rabbit @module=logmon path=/opt/nomad/data/alloc/9ef77798-f8de-f31f-d786-89378fb82b45/alloc/logs/.rabbit.stdout.fifo timestamp=2021-03-13T18:28:57.391+0100
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.392+0100 [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=9ef77798-f8de-f31f-d786-89378fb82b45 task=rabbit path=/opt/nomad/data/alloc/9ef77798-f8de-f31f-d786-89378fb82b45/alloc/logs/.rabbit.stderr.fifo @module=logmon timestamp=2021-03-13T18:28:57.392+0100
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.413+0100 [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=2264563b-b2a6-86fc-0d2c-8f6c193c05f3 task=mode-api @module=logmon path=/opt/nomad/data/alloc/2264563b-b2a6-86fc-0d2c-8f6c193c05f3/alloc/logs/.mode-api.stdout.fifo timestamp=2021-03-13T18:28:57.412+0100
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.415+0100 [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=2264563b-b2a6-86fc-0d2c-8f6c193c05f3 task=mode-api path=/opt/nomad/data/alloc/2264563b-b2a6-86fc-0d2c-8f6c193c05f3/alloc/logs/.mode-api.stderr.fifo @module=logmon timestamp=2021-03-13T18:28:57.414+0100
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.437+0100 [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=1dd29296-2043-7845-7fd9-ad9c6401338d task=mode-api @module=logmon path=/opt/nomad/data/alloc/1dd29296-2043-7845-7fd9-ad9c6401338d/alloc/logs/.mode-api.stdout.fifo timestamp=2021-03-13T18:28:57.437+0100
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.438+0100 [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=1dd29296-2043-7845-7fd9-ad9c6401338d task=mode-api @module=logmon path=/opt/nomad/data/alloc/1dd29296-2043-7845-7fd9-ad9c6401338d/alloc/logs/.mode-api.stderr.fifo timestamp=2021-03-13T18:28:57.437+0100
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.442+0100 [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0e0513cc-1ea3-2b0a-dbaf-14730684a0d4 task=mode-api path=/opt/nomad/data/alloc/0e0513cc-1ea3-2b0a-dbaf-14730684a0d4/alloc/logs/.mode-api.stdout.fifo @module=logmon timestamp=2021-03-13T18:28:57.442+0100
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.444+0100 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=8000bb3712ed59bd61322800f98fff2c2ae7c3fc928a3dd56817c875dbf04e39
Mar 13 18:28:57 server-0 nomad[1092]: 2021-03-13T18:28:57.459+0100 [INFO] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0e0513cc-1ea3-2b0a-dbaf-14730684a0d4 task=mode-api path=/opt/nomad/data/alloc/0e0513cc-1ea3-2b0a-dbaf-14730684a0d4/alloc/logs/.mode-api.stderr.fifo @module=logmon timestamp=2021-03-13T18:28:57.458+0100
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reboot-and-systemd startup sequence described in the issue, comparing it with manual startup. Inspect the supplied Nomad configuration, systemd unit, and logs, especially the advertised and Raft addresses and the stable-configuration warning. Done means identifying the startup/interface cause and documenting or testing a corrective change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100