hashicorp / hashicorp/consul

Consul cluster fails to start after a restart due to /etc/consul.d/server_metadata.json: invalid config key last_seen_unix

Open
#21,364 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

After deploying a 3 node consul cluster successfully, Consul writes out to the file `/etc/consul.d/server_metadata.json` and then complains about the key index `last_seen_unix` after restarting/reloading Consul. The servers in the cluster fail to start up until this file is removed.

Contents of the file:
```
{"last_seen_unix":1719273473}
```

---

#### Reproduction Steps

Steps to reproduce this issue:

1. Create a cluster with 3 server nodes, 0 clients
2. Stop Consul
3. Start Consul
4. They will not start, each one logging the same error

### Operating system and Environment details

Debian 12, Consul 1.18.2 (not running 1.19.0 due to DNS bug with Nomad)

### Log Fragments

```
Jun 24 18:31:43 mynode systemd[1]: Started consul.service - "HashiCorp Consul - A service mesh solution".
Jun 24 18:31:43 mynode consul[3835]: ==> failed to parse /etc/consul.d/server_metadata.json: 1 error occurred:
Jun 24 18:31:43 mynode consul[3835]: * invalid config key last_seen_unix
Jun 24 18:31:43 mynode systemd[1]: consul.service: Main process exited, code=exited, status=1/FAILURE
Jun 24 18:31:43 mynode systemd[1]: consul.service: Failed with result 'exit-code'.
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the restart sequence with a three-node Consul cluster and inspect how /etc/consul.d/server_metadata.json is written and loaded. Trace the configuration parsing path for the last_seen_unix key, then verify that a stop-and-start no longer fails with the reported invalid config key error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.