[BUG]: Error after force restart

What version of Dgraph are you using?

Dgraph version   : v23.1.0
Dgraph codename  : dgraph
Dgraph SHA-256   : 2b0d2fb977807f9d681c3a8e5f67a6fb133c99c772009158107aa6b1ac4cbd10
Commit SHA-1     : 2b18d19
Commit timestamp : 2023-08-17 13:27:10 -0500
Branch           : HEAD
Go version       : go1.19.12
jemalloc enabled : true

Tell us a little more about your go-environment?

No response

Have you tried reproducing the issue with the latest release?

None

What is the hardware spec (RAM, CPU, OS)?

N/A

What steps will reproduce the bug?

Force a restart.

Expected behavior and actual result.

Recover from a restart

Additional information

I0709 16:43:48.665951       1 node.go:312] Found Snapshot.Metadata: {ConfState:{Nodes:[1] Learners:[] XXX_unrecognized:[]} Index:34256070 Term:67 XXX_unrecognized:[]}
I0709 16:43:48.665984       1 node.go:323] Found hardstate: {Term:70 Vote:1 Commit:34318521 XXX_unrecognized:[]}
I0709 16:43:49.168983       1 pool.go:165] CONN: Connecting to alpha1:7080
I0709 16:43:50.369417       1 node.go:328] Group 1 found 88501 entries
I0709 16:43:50.369479       1 draft.go:1803] Restarting node for group: 1
I0709 16:43:50.369506       1 node.go:190] Setting conf state to nodes:1 
2024/07/09 16:43:50 1 state.commit 34318521 is out of range [34256070, 34318501]
panic: 1 state.commit 34318521 is out of range [34256070, 34318501]

goroutine 840 [running]:
log.Panicf({0x20af210?, 0xb41d85?}, {0xc00f7968c0?, 0x1dc7a40?, 0xc00024ec00?})
        /opt/hostedtoolcache/go/1.19.12/x64/src/log/log.go:395 +0x67
github.com/dgraph-io/dgraph/x.(*ToGlog).Panicf(0x20ba8a5?, {0x20af210?, 0x90?}, {0xc00f7968c0?, 0xb44401?, 0x436b7f5c265528f5?})
        /home/runner/work/dgraph/dgraph/x/log.go:39 +0x32
go.etcd.io/etcd/raft.(*raft).loadState(0xc0001ce640, {0x46, 0x1, 0x20ba8b9, {0x0, 0x0, 0x0}})
        /home/runner/go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20190108173120-83c051b701d3/raft/raft.go:1475 +0x1a4
go.etcd.io/etcd/raft.newRaft(0xc01402a000)
        /home/runner/go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20190108173120-83c051b701d3/raft/raft.go:377 +0x791
go.etcd.io/etcd/raft.RestartNode(0xc01402a000)
        /home/runner/go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20190108173120-83c051b701d3/raft/node.go:242 +0x33
github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode(0xc00e87ff00)
        /home/runner/work/dgraph/dgraph/worker/draft.go:1829 +0x6f6
github.com/dgraph-io/dgraph/worker.StartRaftNodes(0xc00010e980, 0x1?)
        /home/runner/work/dgraph/dgraph/worker/groups.go:157 +0x65f
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run.func4()
        /home/runner/work/dgraph/dgraph/dgraph/cmd/alpha/run.go:813 +0x34
created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
        /home/runner/work/dgraph/dgraph/dgraph/cmd/alpha/run.go:812 +0x14f1

Start by reproducing the force-restart failure with Dgraph v23.1.0, then read worker/draft.go around InitAndStartNode and the etcd raft loadState path shown in the stack trace. Compare the snapshot and hardstate indexes in the logs; done means the node recovers from a force restart without the out-of-range panic.