Consul cluster fails to start after a restart due to /etc/consul.d/server_metadata.json: invalid config key last_seen_unix
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
After deploying a 3 node consul cluster successfully, Consul writes out to the file `/etc/consul.d/server_metadata.json` and then complains about the key index `last_seen_unix` after restarting/reloading Consul. The servers in the cluster fail to start up until this file is removed.
Contents of the file:
```
{"last_seen_unix":1719273473}
```
---
#### Reproduction Steps
Steps to reproduce this issue:
1. Create a cluster with 3 server nodes, 0 clients
2. Stop Consul
3. Start Consul
4. They will not start, each one logging the same error
### Operating system and Environment details
Debian 12, Consul 1.18.2 (not running 1.19.0 due to DNS bug with Nomad)
### Log Fragments
```
Jun 24 18:31:43 mynode systemd[1]: Started consul.service - "HashiCorp Consul - A service mesh solution".
Jun 24 18:31:43 mynode consul[3835]: ==> failed to parse /etc/consul.d/server_metadata.json: 1 error occurred:
Jun 24 18:31:43 mynode consul[3835]: * invalid config key last_seen_unix
Jun 24 18:31:43 mynode systemd[1]: consul.service: Main process exited, code=exited, status=1/FAILURE
Jun 24 18:31:43 mynode systemd[1]: consul.service: Failed with result 'exit-code'.
```
Contributor guide
Research direction
Start by reproducing the restart sequence with a three-node Consul cluster and inspect how /etc/consul.d/server_metadata.json is written and loaded. Trace the configuration parsing path for the last_seen_unix key, then verify that a stop-and-start no longer fails with the reported invalid config key error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100