Consul server pod crashes on startup

#### Overview of the Issue

Hi, we are using Consul with 3 server nodes in Kubernetes, after restarting Consul server pods, one pod wont come back with following stack trace:

```
==> Starting Consul agent...
Version: '1.14.7'
Build Date: '2023-05-16 01:36:41 +0000 UTC'
Node ID: '037f79b8-453c-7cde-907e-b92b1873b6ab'
Node name: 'consul-server-2'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, gRPC-TLS: -1, DNS: 8600)
Cluster Addr: 10.128.111.140 (LAN: 8301, WAN: 8302)
Gossip Encryption: false
Auto-Encrypt-TLS: false
HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2
gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2

==> Log data will now stream in as it occurs:

2024-07-15T11:01:30.362Z [WARN] agent: bootstrap_expect > 0: expecting 3 servers
2024-07-15T11:01:30.455Z [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers
panic: runtime error: index out of range [7] with length 4

goroutine 1 [running]:
encoding/binary.bigEndian.Uint64(...)
encoding/binary/binary.go:179
github.com/hashicorp/raft-boltdb/v2.bytesToUint64(...)
github.com/hashicorp/raft-boltdb/v2@v2.2.2/util.go:29
github.com/hashicorp/raft-boltdb/v2.(*BoltStore).LastIndex(0xb?)
github.com/hashicorp/raft-boltdb/v2@v2.2.2/bolt_store.go:150 +0x1ae
github.com/hashicorp/raft.(*LogCache).LastIndex(0xc000e38320?)
github.com/hashicorp/raft@v1.3.11/log_cache.go:72 +0x22
github.com/hashicorp/raft.NewRaft(0xc000e38320, {0x41bdcb0?, 0xc000a8e570}, {0x41ce2e8?, 0xc0013ca900}, {0x41c9988?, 0xc000db79f8}, {0x41bdf80?, 0xc000dc0e70}, {0x41d6a90, ...})
github.com/hashicorp/raft@v1.3.11/api.go:501 +0x1bc
github.com/hashicorp/consul/agent/consul.(*Server).setupRaft(0xc000d3f800)
github.com/hashicorp/consul/agent/consul/server.go:1049 +0x10cc
github.com/hashicorp/consul/agent/consul.NewServer(0xc0004b9200, {0xc00057c0f0, {0x41e6898, 0xc000a8ec00}, 0xc000364280, 0xc000b869a0, 0xc000ba6080, 0xc000cc6000, {0x41ba6e0, 0xc0012c2ae0}, ...}, ...)
github.com/hashicorp/consul/agent/consul/server.go:575 +0x1915
github.com/hashicorp/consul/agent.(*Agent).Start(0xc00103c380, {0x41c9170, 0xc001188000})
github.com/hashicorp/consul/agent/agent.go:607 +0x6bb
github.com/hashicorp/consul/command/agent.(*cmd).run(0xc0008fb600, {0xc0000720d0, 0x9, 0x9})
github.com/hashicorp/consul/command/agent/agent.go:274 +0x13ef
github.com/hashicorp/consul/command/agent.(*cmd).Run(0xc0008fb600, {0xc0000720d0?, 0xffffffffffffffff?, 0x0?})
github.com/hashicorp/consul/command/agent/agent.go:71 +0x27
github.com/mitchellh/cli.(*CLI).Run(0xc000364a00)
github.com/mitchellh/cli@v1.1.0/cli.go:260 +0x5f8
main.realMain()
github.com/hashicorp/consul/main.go:48 +0x40f
main.main()
github.com/hashicorp/consul/main.go:18 +0x19
```

after deleting all data from this Consul node, it starts successfully. We are not able to reproduce this issue.

Consul server config:

```
data:
acl-config.json: |-
{
"acl": {
"enabled": true,
"default_policy": "deny",
"down_policy": "extend-cache",
"enable_token_persistence": true
}
}
central-config.json: |-
{
"enable_central_service_config": true
}
extra-from-values.json: '{"disable_update_check":true}'
server.json: |
{
"bind_addr": "0.0.0.0",
"bootstrap_expect": 3,
"client_addr": "0.0.0.0",
"connect": {
"enabled": true
},
"datacenter": "dc1",
"data_dir": "/consul/data",
"domain": "consul",
"ports": {
"grpc": 8502,
"grpc_tls": -1,
"serf_lan": 8301
},
"retry_join": [
"consul-server.consul.svc:8301"
],
"server": true
}
ui-config.json: |-
{
"ui_config": {
"enabled": true,
"metrics_provider": "prometheus",
"metrics_proxy": {
"base_url": "http://prometheus-server"
}
}
}
```

Consul version: 1.14.7

Start with agent/consul/server.go at setupRaft and trace the call into raft-boltdb/v2 BoltStore.LastIndex, using the reported index-out-of-range panic and the persisted /consul/data directory as the investigation context. Determine what corrupt or unexpected data state causes startup to fail, then verify the diagnosis with a reproducible case or regression test; done means the server no longer crashes on the affected persisted state.

Written by the indexing model from the issue text.