SystemD configuration broken with single-node / dev setups
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
_This issue has already been observed and mentioned in https://github.com/hashicorp/consul/issues/16844. Since it's kind of separate, I think it's useful to outline and track this in a new issue._
Since https://github.com/hashicorp/consul/pull/16845 got merged single-node/dev setups of Consul don't work with systemd
out of the box any more. When running `systemctl start consul` systemd is just stuck and times out. Consul appears to start fine, but just systemd does not know that.
[Online docs](https://developer.hashicorp.com/consul/docs/agent#understanding-the-agent-startup-output) mention that the NOTIFY_SOCKET is only notified when a LAN join is completed and `join` or `retry_join` is set. Single node setups don't have that usually. Maybe it is related to that.
### Workaround
`systemctl edit consul`
```
[Service]
Type=simple
```
---
#### Reproduction Steps
### Consul info for both Client and Server
Consul info
Output from 'consul info' command here
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 192df66a
version = 1.16.0
version_metadata =
consul:
acl = disabled
bootstrap = true
known_datacenters = 1
leader = true
leader_addr = 10.0.2.15:8300
server = true
raft:
applied_index = 18
commit_index = 18
fsm_pending = 0
last_contact = 0
last_log_index = 18
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:525e7b31-91a1-1907-5715-d94bc31c73d2 Address:10.0.2.15:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 2
goroutines = 167
max_procs = 2
os = linux
version = go1.20.4
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 1
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 2
members = 1
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```
HCL config
```
data_dir = "/opt/consul"
server=true
bootstrap_expect=1
```
### Operating system and Environment details
```
cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"
```
### Log Fragments
Consul log
```
Jul 12 12:06:26 ubuntu-jammy systemd[1]: Starting "HashiCorp Consul - A service mesh solution"...
Jul 12 12:06:26 ubuntu-jammy consul[2739]: ==> Starting Consul agent...
Jul 12 12:06:26 ubuntu-jammy consul[2739]: Version: '1.16.0'
Jul 12 12:06:26 ubuntu-jammy consul[2739]: Build Date: '2023-06-26 20:07:11 +0000 UTC'
Jul 12 12:06:26 ubuntu-jammy consul[2739]: Node ID: '525e7b31-91a1-1907-5715-d94bc31c73d2'
Jul 12 12:06:26 ubuntu-jammy consul[2739]: Node name: 'ubuntu-jammy'
Jul 12 12:06:26 ubuntu-jammy consul[2739]: Datacenter: 'dc1' (Segment: '')
Jul 12 12:06:26 ubuntu-jammy consul[2739]: Server: true (Bootstrap: true)
Jul 12 12:06:26 ubuntu-jammy consul[2739]: Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: -1, gRPC-TLS: 8503, DNS: 8600)
Jul 12 12:06:26 ubuntu-jammy consul[2739]: Cluster Addr: 10.0.2.15 (LAN: 8301, WAN: 8302)
Jul 12 12:06:26 ubuntu-jammy consul[2739]: Gossip Encryption: false
Jul 12 12:06:26 ubuntu-jammy consul[2739]: Auto-Encrypt-TLS: false
Jul 12 12:06:26 ubuntu-jammy consul[2739]: ACL Enabled: false
Jul 12 12:06:26 ubuntu-jammy consul[2739]: Reporting Enabled: false
Jul 12 12:06:26 ubuntu-jammy consul[2739]: ACL Default Policy: allow
Jul 12 12:06:26 ubuntu-jammy consul[2739]: HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2
Jul 12 12:06:26 ubuntu-jammy consul[2739]: gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
Jul 12 12:06:26 ubuntu-jammy consul[2739]: Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2
Jul 12 12:06:26 ubuntu-jammy consul[2739]: ==> Log data will now stream in as it occurs:
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.939Z [WARN] agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.939Z [WARN] agent: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.939Z [WARN] agent: bootstrap = true: do not enable unless necessary
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.942Z [WARN] agent.auto_config: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.942Z [WARN] agent.auto_config: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.942Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.943Z [INFO] agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:525e7b31-91a1-1907-5715-d94bc31c73d2 Address:10.0.2.15:>
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.944Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.0.2.15:8300 [Follower]" leader-address= leader-id=
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.944Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: ubuntu-jammy.dc1 10.0.2.15
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.944Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: ubuntu-jammy 10.0.2.15
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.944Z [INFO] agent.router: Initializing LAN area manager
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.944Z [WARN] agent.server.serf.wan: serf: Failed to re-join any previously known node
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.944Z [WARN] agent.server.serf.lan: serf: Failed to re-join any previously known node
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.945Z [INFO] agent.server: Adding LAN server: server="ubuntu-jammy (Addr: tcp/10.0.2.15:8300) (DC: dc1)"
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.945Z [INFO] agent.server.autopilot: reconciliation now disabled
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.945Z [INFO] agent.server: Handled event for server in area: event=member-join server=ubuntu-jammy.dc1 area=wan
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.945Z [INFO] agent.server.cert-manager: initialized server certificate management
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.945Z [INFO] agent: Started DNS server: address=127.0.0.1:8600 network=udp
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.946Z [INFO] agent: Started DNS server: address=127.0.0.1:8600 network=tcp
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.946Z [INFO] agent: Starting server: address=127.0.0.1:8500 network=tcp protocol=http
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.946Z [INFO] agent: started state syncer
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.946Z [INFO] agent: Consul agent running!
Jul 12 12:06:26 ubuntu-jammy consul[2739]: 2023-07-12T12:06:26.946Z [INFO] agent: Started gRPC listeners: port_name=grpc_tls address=127.0.0.1:8503 network=tcp
```
SystemD Log
```
Jul 12 12:06:26 ubuntu-jammy systemd[1]: Starting "HashiCorp Consul - A service mesh solution"...
Jul 12 12:07:57 ubuntu-jammy systemd[1]: consul.service: start operation timed out. Terminating.
Jul 12 12:07:57 ubuntu-jammy systemd[1]: consul.service: Main process exited, code=exited, status=1/FAILURE
Jul 12 12:07:57 ubuntu-jammy systemd[1]: consul.service: Failed with result 'timeout'.
Jul 12 12:07:57 ubuntu-jammy systemd[1]: Failed to start "HashiCorp Consul - A service mesh solution".
```
Contributor guide
Research direction
Start by reproducing the single-node configuration with `systemctl start consul` and inspect the Consul systemd service's startup notification behavior, especially `NOTIFY_SOCKET`. Compare the Consul and systemd log fragments and the documented startup conditions. Done means a single-node or dev setup starts successfully under systemd without the timeout workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100