hashicorp / hashicorp/consul

server does not shutdown cleanly

Open
#21,911 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

When asked by systemd to stop, consul still exits with status 1, tripping over its own shutdown.

---

#### Reproduction Steps

1. `systemctl stop consul.service`

```
systemd[1]: Stopping consul.service...
consul[623754]: 2024-10-28T21:09:51.702Z [INFO] agent.router.manager: shutting down
consul[623754]: 2024-10-28T21:09:51.702Z [INFO] agent.router.manager: shutting down
consul[623754]: 2024-10-28T21:09:51.706Z [INFO] agent: consul server down
consul[623754]: 2024-10-28T21:09:51.706Z [INFO] agent: shutdown complete
consul[623754]: 2024-10-28T21:09:51.706Z [INFO] agent.dns: Stopping server: protocol=DNS address=127.0.0.1:8600 network=tcp
consul[623754]: 2024-10-28T21:09:51.706Z [INFO] agent.dns: Stopping server: protocol=DNS address=127.0.0.1:8600 network=udp
consul[623754]: 2024-10-28T21:09:51.706Z [INFO] agent: Stopping server: address=127.0.0.1:8500 network=tcp protocol=http
consul[623754]: 2024-10-28T21:09:51.789Z [ERROR] agent.server.raft: failed to decode incoming command: error="transport shutdown"
consul[623754]: 2024-10-28T21:09:51.828Z [ERROR] agent.server.raft: failed to decode incoming command: error="transport shutdown"
consul[623754]: 2024-10-28T21:09:52.707Z [WARN] agent: Failed to stop server: address=127.0.0.1:8500 network=tcp protocol=http
consul[623754]: 2024-10-28T21:09:52.707Z [INFO] agent: Waiting for endpoints to shut down
consul[623754]: 2024-10-28T21:09:52.707Z [INFO] agent: Endpoints down
consul[623754]: 2024-10-28T21:09:52.707Z [INFO] agent: Exit code: code=1
systemd[1]: consul.service: Main process exited, code=exited, status=1/FAILURE
```

I find this output very confusing. First it says `agent: consul server down` and `agent: shutdown complete`.
So I assume it's shut down.

But then it prints more things to shut down, including `Stopping server`. Wait, I thought the server was already down as per output above?

Then it notices `transport shutdown`, and apparently considers ihat an error and exits with exit code 1.

But that doesn't make sense, since it's down because we asked it to shut down.

## Environment

Consul `v1.18.2` on Linux; config:

```json
{
"autopilot": {
"min_quorum": 3
},
"bind_addr": "10.0.0.3",
"bootstrap_expect": 3,
"check_update_interval": "1ns",
"data_dir": "/var/lib/consul",
"dns_config": {
"allow_stale": false
},
"enable_debug": true,
"enable_script_checks": true,
"gossip_lan": {
"probe_interval": "1000ms"
},
"performance": {
"raft_multiplier": 1
},
"reconnect_timeout": "8760h",
"reconnect_timeout_wan": "8760h",
"retry_interval": "1s",
"retry_join": [
"10.0.0.1",
"10.0.0.2"
],
"server": true,
"ui_config": {
"enabled": true
}
}
```

## Proposed approach

* [ ] Consul should exit with code 0 when asked to shut down, not treating `transport shutdown` as errors when they are due to the shutdown requested by the user.
* [ ] The logs would ideally make more sense to make clear to the human operator what's already shut down and what isn't.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the shutdown with `systemctl stop consul.service` and tracing the shutdown sequence represented in the issue logs, especially the Raft `transport shutdown` errors and final exit code. Done means a user-requested stop exits with code 0 and the logs clearly distinguish completed shutdown steps from failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
backend, distributed-systems, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.