server does not shutdown cleanly
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
When asked by systemd to stop, consul still exits with status 1, tripping over its own shutdown.
---
#### Reproduction Steps
1. `systemctl stop consul.service`
```
systemd[1]: Stopping consul.service...
consul[623754]: 2024-10-28T21:09:51.702Z [INFO] agent.router.manager: shutting down
consul[623754]: 2024-10-28T21:09:51.702Z [INFO] agent.router.manager: shutting down
consul[623754]: 2024-10-28T21:09:51.706Z [INFO] agent: consul server down
consul[623754]: 2024-10-28T21:09:51.706Z [INFO] agent: shutdown complete
consul[623754]: 2024-10-28T21:09:51.706Z [INFO] agent.dns: Stopping server: protocol=DNS address=127.0.0.1:8600 network=tcp
consul[623754]: 2024-10-28T21:09:51.706Z [INFO] agent.dns: Stopping server: protocol=DNS address=127.0.0.1:8600 network=udp
consul[623754]: 2024-10-28T21:09:51.706Z [INFO] agent: Stopping server: address=127.0.0.1:8500 network=tcp protocol=http
consul[623754]: 2024-10-28T21:09:51.789Z [ERROR] agent.server.raft: failed to decode incoming command: error="transport shutdown"
consul[623754]: 2024-10-28T21:09:51.828Z [ERROR] agent.server.raft: failed to decode incoming command: error="transport shutdown"
consul[623754]: 2024-10-28T21:09:52.707Z [WARN] agent: Failed to stop server: address=127.0.0.1:8500 network=tcp protocol=http
consul[623754]: 2024-10-28T21:09:52.707Z [INFO] agent: Waiting for endpoints to shut down
consul[623754]: 2024-10-28T21:09:52.707Z [INFO] agent: Endpoints down
consul[623754]: 2024-10-28T21:09:52.707Z [INFO] agent: Exit code: code=1
systemd[1]: consul.service: Main process exited, code=exited, status=1/FAILURE
```
I find this output very confusing. First it says `agent: consul server down` and `agent: shutdown complete`.
So I assume it's shut down.
But then it prints more things to shut down, including `Stopping server`. Wait, I thought the server was already down as per output above?
Then it notices `transport shutdown`, and apparently considers ihat an error and exits with exit code 1.
But that doesn't make sense, since it's down because we asked it to shut down.
## Environment
Consul `v1.18.2` on Linux; config:
```json
{
"autopilot": {
"min_quorum": 3
},
"bind_addr": "10.0.0.3",
"bootstrap_expect": 3,
"check_update_interval": "1ns",
"data_dir": "/var/lib/consul",
"dns_config": {
"allow_stale": false
},
"enable_debug": true,
"enable_script_checks": true,
"gossip_lan": {
"probe_interval": "1000ms"
},
"performance": {
"raft_multiplier": 1
},
"reconnect_timeout": "8760h",
"reconnect_timeout_wan": "8760h",
"retry_interval": "1s",
"retry_join": [
"10.0.0.1",
"10.0.0.2"
],
"server": true,
"ui_config": {
"enabled": true
}
}
```
## Proposed approach
* [ ] Consul should exit with code 0 when asked to shut down, not treating `transport shutdown` as errors when they are due to the shutdown requested by the user.
* [ ] The logs would ideally make more sense to make clear to the human operator what's already shut down and what isn't.
Contributor guide
Research direction
Start by reproducing the shutdown with `systemctl stop consul.service` and tracing the shutdown sequence represented in the issue logs, especially the Raft `transport shutdown` errors and final exit code. Done means a user-requested stop exits with code 0 and the logs clearly distinguish completed shutdown steps from failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- backend, distributed-systems, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100